Multiple Sends Using Multiple Slots In tcp-preload-slot.c

[bornlibra23]

Hi Can anyone help me to use multiple slots in tcp-preload-slot.c
I have tried loading multiple slots & triggering them but to no avail.
Primary problem is that the tcp sequence numbers arent incremented.
the *send_advance function isnt relevant here I think.

Anyone?

[bornlibra23]

Hi its been a long time & no one has responded.
Is anyone there? Or am I talking into the ether? :)

[bornlibra23]

@abbradar @mgrosvenor @ahsanmirzab @ido @fluffy

Anyone?

[sirgajelot]

With the preload, you are loading the entire ethernet frame, so if you want to achieve the best latency but have a bunch of frame options preloaded, then you need to update all of the slots every time you trigger one of them, or at least update the tcp sequence number and checksum for each slot that is still relevant. In my experience it is easier to just reload all of the slots

[bornlibra23]

Thank goodness. Finally someone responded.
I have just achieved this today around an hour earlier.
Here are the issues:

1. The tcp sequence number patching needs to be done manually. This should be done by the library just as its done in non preload case.
2. Each slot need to be trigger manually. Is there a way to trigger them all together?

Who is the owner of the repo to whom I can contact?

[sirgajelot]

It's owned by Cisco but good luck getting much response. I have learnt everything I know from studying the example source and reading the online docs. I don't know whether any of the people you are asking still work as Cisco. I know Matt Grosvenor moved to Jump, for instance

[bornlibra23]

Can you help me here? May be issue 2 as issue 1 seems more involved.

[mgrosvenor]

Hi team,
This is a Cisco product. So the best bet for support is to go through your Cisco TAC account and make a support ticket. As Andy mentioned, I work for Jump now so it's not clear if I'm even allowed to do anything here, probably not.

Best,
Matt

[sirgajelot]

If the messages are all going to the same socket, then it will pay to combine the TCP payloads into as few frames as possible. If these are all going to different sockets, then will have independent sequence numbers, so your first problem does not exist. There is no way to fire all of the buffers in one instruction but in practice, you can write to TX_COMMAND multiple times in a row and the buffers should all go out in sequence.

Note this from exanic-tx-preload.c
/*
* Trigger a send from each slot spaced 1 sec apart.
* Note: To avoid overflowing the TX command queue, the software should make
* sure that no more than buffer_size/EXANIC_TX_CMD_FIFO_SIZE_DIVISOR
* transmit commands are in flight at any one time. The TX feedback
* mechanism (not used in this example) can be used to notify the software
* when a transmit command is finished.
*/
for (i = 0; i < 4; i++)
{
trigger_slot_send(tx, i);
usleep(1000000);
}

It is important that you pay attention to the buffer_size/EXANIC_TX_CMD_FIFO_SIZE_DIVISOR value. To do this you can set feedback, so that you know when an entry has been cleared from the transmit queue.

[bornlibra23]

Hi team, This is a Cisco product. So the best bet for support is to go through your Cisco TAC account and make a support ticket. As Andy mentioned, I work for Jump now so it's not clear if I'm even allowed to do anything here, probably not.

Best, Matt

Okay but cant anyone patch this open source code? I assume you are possibly under some NDA which disallows you from working on this code?

I dont have a Cisco TAC account. I am just using this as an opensource software.

[sirgajelot]

For what it's worth, we normally just work off a fork in my company. If there is something that we think is worthwhile for the general community, then we might open an Issue but it is up to the maintainers to adopt any suggestions.

[bornlibra23]

If the messages are all going to the same socket, then it will pay to combine the TCP payloads into as few frames as possible. If these are all going to different sockets, then will have independent sequence numbers, so your first problem does not exist. There is no way to fire all of the buffers in one instruction but in practice, you can write to TX_COMMAND multiple times in a row and the buffers should all go out in sequence.

I am only expecting the same functionality as normal accelerated socked does, the one which uses tcp_send_advance in the end to maintain tcp state.

So even if the packets are destined for different sockets it should should work ideally.

And yes I am looping on TX_COMMAND currently.

[sirgajelot]

This function is not designed to give you the same function as an accelerated socket. It is there to optimise latency as much as possible. If you want the flexibility of something like accelerated sockets, then you should use exasock functions or simply use the exasock LD_PRELOAD method

[bornlibra23]

This function is not designed to give you the same function as an accelerated socket. It is there to optimise latency as much as possible. If you want the flexibility of something like accelerated sockets, then you should use exasock functions or simply use the exasock LD_PRELOAD method

Thats what I used first. It was slower. I switched to using exanic functions for speed [exasock_tcp_build_header, exasock_tcp_set_length, exasock_tcp_calc_checksum, exanic_transmit_frame, exasock_tcp_send_advance].
Now I want to use preload functionality to go even faster.

The business use case is to send thousands of orders to the exchange when it opens.

EDIT: Sorry my mistake by accelerated socket I didnt mean the one under preload. I meant generically "faster sockets" My mistake.

By the way if anyone wants to work with me as a paid freelancer on this, I will be open to that.

[mgrosvenor]

Hi team, This is a Cisco product. So the best bet for support is to go through your Cisco TAC account and make a support ticket. As Andy mentioned, I work for Jump now so it's not clear if I'm even allowed to do anything here, probably not.

Best, Matt
Okay but cant anyone patch this open source code? I assume you are possibly under some NDA which disallows you from working on this code?

I dont have a Cisco TAC account. I am just using this as an opensource software.

The code might be open source but the owner of the project is Cisco. If you want it patched then you’ll need Cisco to allocate someone to fix it and that comes with a support cost. Unfortunately everyone I knew at Cisco has left or been fired so I can’t point you at anyone internal other than going through the offical TAC front door.

In theory you could pay someone like me to do the work for you, but since most users of this product work for the competition of the company I work for I’d be legally prohibited from doing it.

On the other hand, you’re most welcome to make a fork and patch it yourself. And if you want to, contribute those patches upstream. Though given Cisco’s apparent lack of interest it seems that is unlikely to go anywhere.

Best,
Matt

[bornlibra23]

In theory you could pay someone like me to do the work for you, but since most users of this product work for the competition of the company I work for I’d be legally prohibited from doing it.

I understand.

On the other hand, you’re most welcome to make a fork and patch it yourself. And if you want to, contribute those patches upstream. Though given Cisco’s apparent lack of interest it seems that is unlikely to go anywhere.

I tried understanding but couldnt. Can anyone give me some pointers? Currently I just want the TCP sequence numbers to be patched internally with every packet I write to the card.