Pick 4 random points A, B, C, D in a plane (in a rectangle, say) and draw the tetragon A-B-C-D-A.

Note that the definition of "convex" here is non-standard.  A convex
polygon is normally defined to be one whose angles are all less than
180 degrees, or, equivalently, one for which all of the diagonals are
inside the polygon.  For example, the quadrilateral with vertices
(1,1), (0,0), (1,-1), (-1,0) is concave.  I'll use your definition for
the analysis.

>My suggestion for "random" tetragons would be: Pick 4 randomly chosen
>points A,B,C,D in a plane (a rectangle, e.g.) and draw the tetragon
>A-B-C-D-A.

If the convex hull of the vertices is a triangle (i.e., one vertex is
inside the triangle given by the other three), then no two edges can
intersect, regardless of the order of the vertices.  If the convex hull
is a quadrilateral (so that the points define a convex quadrilateral),
the diagonals intersect, and the probability is 2/3 that the diagonals
will be edges of the tetragon.  (For example, if ABCD is a convex
quadrilateral, then AC and BD intersect, so ABDC and ACDB have
intersecting edges.)

Thus we want to know the probability that one point is contained in
the triangle defined by the other three.  This is four times the
probability that D is contained in the triangle ABC, since it can
happen with only one point for any choice of four.

If D has the smallest or largest x-coordinate of the four points
(probability 1/2), or the smallest or largest y-coordinate (probability
1/2), it cannot be inside the triangle.  The probability that one of
these cases happens is 3/4.  Thus, only 1/4 of the time is what follows
possible.

Now re-scale the coordinate system so that the smallest x-coordinate
and y-coordinate of A, B, and C are both 0, and the largest are 1.
Thus we know that D is somewhere in the unit square, and the
probability that D is in the triangle is equal to the area of the
triangle.

The probability is 1/3 that the two points with x-coordinates 0 and 1
also have y-coordinates 0 and 1.  If this happens, then we have, say,
that A is (0,0), B is (x,y) with x<y, and C is (1,1).  The base AC of
this triangle is sqrt(2), and its altitude is (y-x)/sqrt(2), so its
area is (y-x)/2.  Since the area of the region which can contain B is
1/2, the average area of the triangle is twice the integral of (y-x)/2
over the region; that is (TeX style follows:)

\int_0^1 \int_0^y (y-x) dx dy =
\int_0^1 (xy - x^2/2)|_0^y dy =
\int_0^1 y^2/2 dy             =
1/6.

If the case in the last paragraph doesn't happen, then we have, say,
that A is (0,0), B is (x,1), and C is (1,y).  The resulting triangle
can be split into three pieces by letting P be the point (x,y) and
considered PAB, PAC, and PBC.  PBC has area (1-x)(1-y)/2, since it is
half a rectangle.  PAB has base PB of length 1-y and altitude x, so its
area is x(1-y)/2; likewise, PAC has area (1-x)y/2.  Thus the total
area is (1-xy)/2.  To get the average, we integrate this over the whole
square, since x and y can have any value from 0 to 1.  The integral of
1/2 over the square is obviously 1/2; for the other term:

\int_0^1 \int_0^1 xy/2 dy dx   =
\int_0^1 (x/2) (y^2/2)|_0^1 dx =
\int_0^1 (x/4) dx              =
1/8.

Thus, in this case, our triangle has expected area 3/8.  (This can also
be seen without integration; each of xy/2, (1-x)y/2, x(1-y)/2, and
(1-x)(1-y)/2 has the same expected value, and their sum is 1/2, so the
sum of three of them must have expected value 3/8.)

So the expected area of the triangle has probability 1/3 of being 1/6,
and probability 2/3 of being 3/8, for a total expected area of
1/18+1/4=11/36.  This is the probability that D is in the triangle,
given that it is in the unit square, so the total probability that D
is in the triangle is 11/144.

Multiplying this by 4 gives a probability 11/36 that the convex hull is
a triangle, and thus 25/36 that it is a quadrilateral.  If it is a
quadrilateral, the probability that two edges of the tetragon intersect
is 2/3, so the total probability that two edges of the tetragon
intersect is 25/54=.46296...  That's less than 1/2, not the more than
1/2 you suspected, but it's close.

-- David Grabiner <grabiner@math.harvard.edu> (January 1995)