This should be fun. AE CS3 includes a new sampleImage() function that not only allows you to sample a layer's pixel color data, but a pixel's alpha value as well. This opens up a previously-impossible opportunity to create an expression that can detect when the visible area of a layer is in contact with the visible area of another layer. This is precisely the capability we need to design a collision detection expression. While we're at it, we might as well go all the way and design it to work even if the layers are scaled, rotated, and/or parented.

The basic idea is that we're going to use sampleImage() to look for a pixel that has a non-zero alpha value for both the layer with the expression and any other layer in the comp. We only have to find one such pixel to determine that a collision has occurred. The obvious, brute-force method of doing this would be to use sampleImage() to retreive the alpha value of every pixel in the layer and compare it to the same world space position in every other layer. That method would be ridiculously slow, so we have to be smarter.

What we need to do is limit our search to the areas where our layer actually overlaps other layers. If we didn't have to worry about a layer being rotated, we could just use the upper left and lower right corners of each layer to determine if/where they overlap. Since we want to be able to accommodate rotated layers, we need to add an extra step and calculate each layer's bounding box. The diagram below should illustrate what we mean by "bounding box".

Now let's look at what happens when two layers intersect. We can limit our search to the relatively small rectangular area of intersection outlined in yellow in the illustration below.

Now our process is starting to take shape. It would be really nice if there was a built-in way to get a layer's bounding box, but there isn't, so we'll have to make our own. Once we calculate the bounding box of our layer, then for each other layer in the comp (until we detect a collision or run out of layers to check) we need to calculate its bounding box, calculate the bounding box of the intersection, and then search the intersection for pixels where the alpha value is non-zero for both layers.

To summarize, we need to create a piece of code that will calculate the bounding box of a layer, another piece to calculate the intersecting rectangle of two overlapping bounding boxes, and a piece to examine each pixel in the intersecting area until a pixel with non-zero aplha in both layers is found. The expression required to do these things is a little lengthy, but it's fairly straight forward. To accommodate layer scaling, rotation, and parenting, we just need to make sure that we do our calculations in the world space coordinate system. So you'll notice the use of the toWorld() and fromWorld() layer space transforms to get the corners of our bounding boxes into and out of world space coordinates.

There's no point in detecting a collision unless you do something when it happens, right? For the purposes of this demonstration, we'll just set the layer's opacity to 100% if it's touching any other layer and 30% if it isn't. OK - it's time to take a look at this monster.

function getMin(a, b, c, d){
  return Math.min(Math.min(a,b),Math.min(c,d));
}

function getMax(a, b, c, d){
  return Math.max(Math.max(a,b),Math.max(c,d));
}

function getBoundingBox(theLayer){
  bb = [];
  c1 = theLayer.toWorld([0,0]);
  c2 = theLayer.toWorld([theLayer.width,0]);
  c3 = theLayer.toWorld([theLayer.width,theLayer.height]);
  c4 = theLayer.toWorld([0,theLayer.height]);
  bb[0] = getMin(c1[0],c2[0],c3[0],c4[0]);
  bb[1] = getMin(c1[1],c2[1],c3[1],c4[1]);
  bb[2] = getMax(c1[0],c2[0],c3[0],c4[0]);
  bb[3] = getMax(c1[1],c2[1],c3[1],c4[1]);
  return bb;
}

function checkLayers(){
  cUL = [];
  cLR = [];
  for ( idx = 1; idx <= thisComp.numLayers; idx++){
    if (index == idx) continue;
    L = thisComp.layer(idx);
    if (! L.active) continue;
    BB = getBoundingBox(L);
    UL = [BB[0],BB[1]];
    LR = [BB[2],BB[3]];

    if (!(myLR[1] < UL[1] || LR[1] < myUL[1] ||
          myLR[0] < UL[0] || LR[0] < myUL[0])){

      cUL[1] = (myUL[1] < UL[1]) ? UL[1] : myUL[1];
      cUL[0] = (myUL[0] < UL[0]) ? UL[0] : myUL[0];
      cLR[1] = (myLR[1] < LR[1]) ? myLR[1] : LR[1];
      cLR[0] = (myLR[0] < LR[0]) ? myLR[0] : LR[0];

      for(i = cUL[0]; i <= cLR[0]; i++){
        for (j = cUL[1]; j <= cLR[1]; j++){
          if (sampleImage(fromWorld([i,j]))[3] > 0 &&
              L.sampleImage(L.fromWorld([i,j]))[3] > 0){

            return true;
          }
        }
      }
    }
  }
  return false;
}

myBB = getBoundingBox(thisLayer);
myUL = [myBB[0],myBB[1]];
myLR = [myBB[2],myBB[3]];

if (checkLayers()){ 100 }else{ 30 }

Collision detection expression allows each layer to detect when its visible area is in contact with the visible area of any other layer.

It's important to note that a bounding box can be completely defined by establishing two diagonal corners. In our case, we'll use the upper left corner (which we'll abbreviate "UL") and the lower right corner ("LR").