[En-Nut-Discussion] TCP vs. Memory - a solution

[Michael Jones]

Hi,

This, proof of concept change seems to work:

Tcpsm.c, NutTcpStateEstablished, 1190

    if (htonl(th->th_seq) > sock->so_rx_nxt) {

        fputc('!', stdout);  //@DEBUG

        NETBUF *nbq;
        NETBUF **nbqp;
        TCPHDR *thq;
        uint32_t th_seq;
        uint32_t thq_seq;

        uint16_t Count = 0;  // <----
        uint32_t Size = 0;   // <----

        if (nb->nb_ap.sz) {
            nbq = sock->so_rx_nbq;
            nbqp = &sock->so_rx_nbq;
            while (nbq) {

                Count++;			// <----
                Size += nbq->nb_ap.sz;	// <----

                thq = (TCPHDR *) (nbq->nb_tp.vp);
                th_seq = htonl(th->th_seq);
                thq_seq = htonl(thq->th_seq);
                if (th_seq < thq_seq && nb->nb_ap.sz <= thq->th_win) {
                    *nbqp = nb;
                    nb->nb_next = nbq;
                    break;
                }
                if (th_seq == thq_seq || nb->nb_ap.sz > thq->th_win) {
                    NutNetBufFree(nb);
                    sock->so_tx_flags |= SO_ACK | SO_FORCE;
                    NutTcpOutput(sock, 0, 0);
                    return;
                }
                nbqp = &nbq->nb_next;
                nbq = nbq->nb_next;
            }
            if (nbq == 0) {
                    
                printf("[%d, %ld]", Count, Size);

                /*
                 * Limit the number of packets in the look-a-head list.
                 * 
                 * Using a constant number of packets is only a hack and
won't be
                 * sufficient in different environments.
                 * 
                 * Alternatives:
                 * 
                 * a) the size of stored packets vs free memory
                 * b) the size of stored packets vs window size
                 * c) ???
                 * 
                 * Also, Size does not reflect the effectively allocated
amount of memory,
                 * as all packets smaller 67 are actually stored as 67
bytes.
                 * 
                 */

                if(Count > 20) {                              
                    NutNetBufFree(nb);
                    sock->so_tx_flags |= SO_ACK | SO_FORCE;
                    NutTcpOutput(sock, 0, 0);
                    return;
                }

                *nbqp = nb;
                nb->nb_next = 0;
            }
        } else
            NutNetBufFree(nb);

        sock->so_tx_flags |= SO_ACK | SO_FORCE;
        NutTcpOutput(sock, 0, 0);
        return;
    }




-----Original Message-----
From: en-nut-discussion-bounces at egnite.de
[mailto:en-nut-discussion-bounces at egnite.de] On Behalf Of Michael Jones
Sent: Donnerstag, 15. Oktober 2009 23:16
To: 'Ethernut User Chat (English)'
Subject: [En-Nut-Discussion] TCP vs. Memory (RFC)

Hi,

I've been spending the last days pulling out my already sparse hair over
this issue.

The environment is a ETHERNUT2 compatible board which receives data from a
server. This server sends bursts of information in packets between 8 to 200
bytes. These bursts might contain between 1 to 1000+ packets. Using
Wireshark I can see 800+ sent within one second. The Nut/OS has about 27000
Bytes free memory when idle (which equals about 300 minimal size (19 + 67)
netbuf allocs).

Speed is not so much a problem, it's totally OK that the board is not able
to handle the load at the speed that it is sent. 

Now to the observations:

1)  The SMCS LAN91C111 will start losing packets once the OS loses the race
to pick the packets up in time.
    (The chip only has 4 packets buffer, so this is by design and too be
expected).

2)  The lost packets cause the "later" packets to be queued into so_rx_nbq
for "future" use.
    (TCPSM.c, NutTcpStateEstablished, ~1190)

3)  The Memory fills up with these future use packets.
    (AFAICS the windows size is not adjusted and there is no limit on these
packets)

4a) The system finally recovers the missing packet and recovers from above
situation.

4b) The future packets use up all free memory till no more packets can be
received.
    (lanc111.c, NicGetPacket, ~1065)

5)  TCP State Machine finally gets fed up and timeouts the Socket and
pending e.g. NutTcpReceive.
    (haven't found the exact point where it does this yet - but it does)

6)  the App calls NutTcpCloseSocket.

7)  NutTcpCloseSocket calls NutTcpStateCloseEvent.

8)  NutTcpStateCloseEvent sets state to TCPS_FIN_WAIT_1.

9)  The state TCPS_FIN_WAIT_1 causes NutTcpOutput(sock, 0, 0) with SO_FIN
and SO_ACK.

9a) NutTcpOutput sends packet to host, 
    host answers causing NutTcpStateProcess to call NutTcpStateFinWait1,
    which calls NutTcpDestroySocket,    
    which frees all used memory and system recovers...

9b) NutTcpOutput FAILS to send packet to host because there is no memory
left,
    Host never answers,
    Memory never gets freed,
    System dies horrible death...

Now there are two aspects in this:

a) How to stop the packets flooding the memory.
b) How to assure that the Socket gets destroyed even in this grim situation.

Comments and thought are welcome.

Cu,
Michael



_______________________________________________
http://lists.egnite.de/mailman/listinfo/en-nut-discussion