This document is Dmitry’s personal attempt to learn TWS API and 99.9% of its content came from TWS API Group https://groups.io/g/twsapi. I tried to structurize some questions and answers in form of FAQ grouped by topics.

Scope

I tried to collect all the general Q/A’s about data flow, tricks of certain functions of TWS API, limitations, workarounds and other general-level info. On the lower-level side of things (language specific) I collected mostly C++ / Java related topics and did not store much on topic of Windows Sockets, ActiveX, CSharp, VB or DDE for Excel etc. Want to add missing topics? You’re welcome to participate and add it yourself! 🙂

In Dec-2014 I started digging (reading through) all the history starting from messages around 05-Feb-2010 and this FAQ is an ongoing attempt to keep track on all important topics.

Disclaimer

This FAQ represents a small fraction of the TWS API Group discussions and by no means is intended to replace one. Please search online group archives for many more answers (than you can find here).

Link to the latest IB API documentation

[A] by Josh

On Wed, May 3, 2017:

The newer documentation is at: http://interactivebrokers.github.io/tws-api/

by the way.

Josh

–also–

———- Forwarded message ———-

Date: Thu, 18 May 2017 17:29:38

From: jbeacomib <notifications@github.com>

Reply-To: InteractiveBrokers/tws-api

To: InteractiveBrokers/tws-api <tws-api@noreply.github.com>

Subject: gh tws-api] Stale IB doco relative to the github content (#493)

The documentation which is actively updated with the most

current information about the builds of TWS and API versions

that are on the website (API versions 9.72, 9.73) is hosted on

[Github](http://interactivebrokers.github.io/tws-api/)

The older documentation was created for version 9.71 of the

API and previous builds of TWS. It is now no longer updated

and has been superseded by the Github documentation site. The

new documentation is correct that up to 32 API applications

can now be connected to one instance of TWS or IB Gateway.

However in practice, this limit is uncommonly reached as it is

usually more efficient to reduce overhead with a smaller

number of connected clients.

TWS app related

[Q] How to turn off Security Code for TWS real account login?

[A] View/Reply Online (#768)

Added on 09-Jul-2020

There is a setting for this at online Account Management. Follow this:

Online account → Settings → User Settings → Secure Login System → Secure Login Device Opt Out → tick the box at “I only want to use my Secure Login Device when logging into Account Management”.

Then press "continue". You will be presented with some disclaimer forms which you have to acknowledge.

If you later change your mind you are able to enable 2FA again, using the same procedure.

I hope this helps.

[Q] How many market data lines one has subscribed to in TWS?

[A] View/Reply Online (#43856)

Added on 23-May-2020

Press Ctrl-Alt-= simultaneously on your keyboard, TWS will show pop-up window with number of lines used.

[Q] How to import Tickers from a File?

[A] discussed here (View/Reply Online (#43800))

https://www.interactivebrokers.com/en/software/tws/usersguidebook/getstarted/import_tickers_from_a_file.htm Here's copy-pasted article from IB's website in case the link above no longer works:

Import Tickers from a File

You can populate a trading page with market data that you import from a comma-delimited .csv file or .txt file. Create the list using any word processing program and save the file with a .txt file extension, or in MS Excel and save as a .csv file.

To create file of ticker symbols to import

Create a file with the extension .txt, for example twsTickers.txt, or create a .csv file in Excel.
In the text file, create one comma-delimited line for each line of market data you want using the syntax described below. In Excel, create one row, using one cell for each field. Note that the first field designation is the line type: DES, SYM, and CON.

DES, UNDERLYING, SECTYPE, EXCHANGE, LASTTRADINGDAY, STRIKE, PUT/CALL, MULTIPLIER

SYM, SYMBOL, EXCHANGE

CON, CONID, EXCHANGE

The DES line type requires information or placeholders for information, with the strike, Put/Call and multiplier fields being optional. Format for the Last Trading Day is YYYYMM, or YYYYMMDD. All entries must be in caps. An example for an options contract XYZ would be:

DES, XYZ, OPT, ISE, 201509, 75, CALL,

A stock instrument for symbol XYZ in this line type would look like this:

DES, XYZ, STK, SMART,,,,

where the commas are placeholders for fields you don't define for stocks.

Using the SYM line type for a stock would look like this:

SYM, XYZ, SMART

Use the CON type to quickly enter contracts with a conid, for example

CON, 12348765, SMART

To import a file into Classic TWS

From the File menu, select Import/Export and then select Import Contracts.
Use the Browse button to find the file to import.
Specify where you want imported tickers to go, either onto an existing page or onto a new page.
Click OK.

To import a file into a Mosaic Watchlist

1. Right click in the Watchlist, and select Import/Export then Import Contracts.

2. Use the Browse button to find the file to import.

3. Click OK.

Imported contracts entered onto an existing page appear at the bottom of the active trading page or Watchlist.

[A] by Josh

typing a local symbol directly into an watchlist row will bring it up immediately, because it is unique to that option

i.e. 'AAPL 200221C00322500' (make sure you have the correct spacing)

Option Symbology Initiative (OSI) standard: https://ibkr.info/article/972

[Q] Link to the official server reset time?

[A] by Josh

https://www.interactivebrokers.com/en/index.php?f=2225

[Q] How to check and solve connectivity issues affecting the Trader Workstation (TWS)

[A] by Josh

https://ibkb.interactivebrokers.com/article/2807

(I’ll preserve a quote from the link above just in case here;)

Background:

The Trader Workstation (TWS) software needs to connect to our gateways and market data servers in order to work properly. Connectivity issues affecting your local network or your Internet Service Provider network may negatively affect the TWS functionality. In this article we will indicate how to test your connectivity using an automated connectivity test web page.

How to test the connectivity using the automated "IB Connectivity Test" web page?

1) Click on this link: http://www.interactivebrokers.com/cgi-bin/conn_test.pl

2) Please wait until all the tests have been completed and results have been displayed. If you see "Success" as outcome for all tests, your connectivity to IB Servers is reliable at the present moment. No additional connectivity troubleshooting or configuration should be needed.

3) If you see "Fail" as outcome for one or more test/s, please click on the link "Fail" itself in order to display the "IB Network Troubleshooting Guide". That section will help you conduct some manual tests to identify the cause of the failure.

Note for Corporative environments and Proxy server users: the automated "Connectivity Test" page may return misleading results in case your machine is accessing the Internet through a Proxy server. This usually happens if you are connected to a company network. If this is your case, we kindly ask you to contact your Network Administrator or your IT Team and ask them to perform a manual connectivity tests towards the destination servers indicated in the table on the top of the IB automated "Connectivity Test" web page itself. The manual connectivity test should be conducted using destination TCP ports 4000 and 4001. Should they prefer to have the server list in another format or should they need to set up the firewall / IP Management rules, you can forward them this page.

(end of quote)

[Q] How to run 2 TWS on different machines (to access same account)?

This question was posted on github ib-controller project:

Hi!

Any chance you can add a setting to deal with the "someone has logged in with this username from another location" scenario?

IB's "great" monster doesn't let you log in from two locations so I'd like to be able to login from my phone to check my portfolio at times that I know the TWS API will not be sending orders.

I would imagine it would click the button after some timeout.

Let me know what you think!

Thanks

Georgios

Added on 16-Aug-2016

[A] by Ben Alex

One option is a second username on your account. Full info at http://ibkb.interactivebrokers.com/node/1004.

[Q] How to revert installed version of API properly? aka downgrading API

Added on 29-May-2015

[A] by Richard L King (found here)

Well, I’ve discovered why I was unable to get any previous ActiveX API version working after uninstalling 9.72 Beta..

When you install the TWS API, the install process puts a library called TwsSocketClient.dll into the Windows system folder. This library has a file version number the same as the API version, and the ActiveX control (Tws.ocx) is actually a wrapper around this library, so Tws.ocx and TwsSocketClient.dll need to be the same version for the ActiveX API to be guaranteed to work properly.

The problem arises from the fact that when you uninstall the API, the TwsSocketClient.dll is not removed. If you now try to install an earlier version of the API, the existing TwsSocketClient.dll is not replaced, presumably because the version number being installed is lower than the one already there. So now the version installed does not match what the ActiveX control expects, and so it doesn’t work – at all.

So the solution is simply to manually delete TwsSocketClient.dll before attempting to install an earlier API version.

rholowczak, perhaps it would be useful including this info in your ActiveX tutorial? Most people probably never revert to a previous API version, but they may want to if, for example, they’ve been trying out a beta version and want to revert to the latest supported version.

[A2] by newguy@videotron.ca

Added on 30-May-2015

That's actually documented by IB.

Uninstalling and Re-installing the TWS API Software on Windows


	Uninstalling and Re-installing the TWS API Software on Windows Open topic with navigation You are here: Overview > Uninstalling and Re-installing on Windows Uninstalling and Re-installing the TWS API Software on Windows
	View on www.interactivebrokers.com

[Q] How to optimize TWS Java VM performance? aka the "latest and greatest" JVM server settings

Added on 27-Apr-2015

[A] by FutureScalper (found here)

you should use IBController by Richard L King

I would emphasize that the most significant thing you can do to improve

performance is to use the Java Server VM. I have a high performance

multi-threaded Swing-based GUI applications which does realtime analysis and

order processing, etc. It runs using the Server VM, even though it is a

"client" application. I also run TWS using the Server VM. This provides a

range of tuning options and global optimization capabilities which probably

doubles or triples throughput.

You can get the Server VM only by installing the JDK, and then you may want

to move the jvm.dll into your JRE area or whatever allows you to use the

command "java -server -version" to verify its installation.

Again, running TWS itself under the Server VM as well is critically

important for high performance.

Here is an example of the command line which I use to run standalone TWS

under the Java Server VM on Windows. Various options are technical, and

subject to change.

c:\WINDOWS\system32\java.exe -server -dsa -Xbatch -Xss192k -Xms200m -Xmx200m

-Xnoclassgc -XX:+ForceTimeHighResolution -XX:-TieredCompilation

-XX:+RelaxAccessControlCheck -XX:MaxInlineSize=64000

-XX:-DontCompileHugeMethods -XX:CompileThreshold=250 -XX:CICompilerCount=1

-XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=150 -XX:+DisableExplicitGC

-XX:ThreadStackSize=192 -XX:+AggressiveOpts -cp

jts.jar;pluginsupport.jar;jcommon-1.0.0.jar;jfreechart-1.0.0.jar;jhall.jar;other.jar;riskfeed.jar;rss.jar

jclient/LoginFrame \jts

The Server VM will compile "huge" methods, and will "inline" code, and

perform global optimizationsn not possible with the standard Client VM.

Performance and scalability matter for many applications.

[Q] How to make TWS application running 24/7 automagically?

[A] you should use IBController by Richard L King

URL: https://github.com/ib-controller/ib-controller

You can download the latest version from
https://github.com/ib-controller/ib-controller/releases/latest

(quote-1)

One possibility is to use IBController, which is a Java program that loads

TWS and intercepts various window events and handles them automatically.

Note that IBController can only handle logon if you have opted out of the

security device scheme.

You can find IBController in the Files section of this forum, under Auto

Richard

(end of quote-1)

(quote-2)

IBController is a Java application that was written to enable Interactive
Brokers Trader Workstation (TWS) to be run in "hands-free" mode. This
makes writing unattended automated trading systems possible. IBController
automates the TWS login by filling the login dialog with your login
credentials. It also handles the dialog boxes that TWS presents
during programmatic trading activies.

(end of quote-2)

link: http://sourceforge.net/projects/ibcontroller/files/

few random notes from IBController users:

—————————————————————————————————————————————-

by Mike Smith mikesmithv@gmail.com

You might try changing the MaxPerSize parameter to something larger, like from 128M to 255M. When that number is too small it can cause any number of weird symptoms. When there is not enough space to load a new class then a previously loaded class is unloaded and strange things can happen. Programmers sometimes assume static class variable are "global variables" that remain as long as the application is launched and that's not the case if the class is unloaded.

Not long ago TWS used the default MaxPerSize and that was not enough if launching under IBController. Adding -XX:MaxPermSize=128M the the IBController launch command was needed because of the extra classes loaded by IBController. Now I see on IB's web site the launch command for TWS now contains XX:MaxPermSize=128M just for a normal TWS launch. So it appears they have added more classes to TWS so they had to bump up that number for a standard launch. So it makes sense a larger number is needed for launching with IBController now.

I don't use IBController anymore so all this is based on my past experiences with it. Now I use the gateway restarting it once a week and just let it run all week on one log in. So all this "in theory" and not my direct experience. I hope it makes sense.

Optimal memory setting

https://groups.io/g/twsapi/topic/4047139

—————————————————————————————————————————————-

[Q] How to set up IBController (3.2.0) + TWS (build 956) on headless Ubuntu 16.04 LTS to run TWO accounts (paper + real) in 10 minutes?

[A] https://dimon.ca/how-to-setup-ibcontroller-and-tws-on-headless-ubuntu-to-run-two-accounts/

Added on 13-Jun-2016

Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.

[Q] How to setup IBC in docker

[A] by dvasdekis <notifications@github.com> view it on GitHub

Added on 19-Jul-2020

I maintain an updated docker container with IB Gateway v978.2c and IBC v3.8.2 here. Feel free to link to it instead of maintaining an independent solution.

[Q] Where do we get the latest versions of TWS API?

[A] http://interactivebrokers.github.io/

[Q] How to reset TWS connections?

[A]

Ctrl-Alt-R resets the account server connection

Ctrl-Alt-F resets the market data connections

[Q] which version of TWS should I use?

[A] by Ruediger Meier sweet_f_a@gmx.de

Actually I install all TWS versions in parallel just to be able to try

it out whenever I will be in the right mood.

The default one which I also use for production is the stable version

936.9l.

For those who don't know it, IB provides stable and latest versions. The

current stable TWS gets only bugfix updates but no new features. That's

why it has such a high minor version number (letter).

[Q] How to reset a paper account?

[A1] View/Reply Online (#46218)

Added on 25-Jan-2021

This, from IB, worked:

You can reset your paper account equity to a value different than the original and up to five times your production account value. Please note that reset requests should be entered before 16:00 EST in order to take effect for the next business day.

Please follow below steps to reset paper equity balance:

1. Log into your paper trading Client Portal by clicking the toggle on the login screen

2. Select Settings followed by Account Settings

3. Click the Configure (gear) icon next to Paper Trading Account Reset

4. Select the reset amount from the drop down menu provided and click Continue.

[A2 (oder – from around "before 2017")] by Eric J. Holtman

Under "Trading Access", pick "Paper Trading Account Reset".

— or —

Log into paper account management (don't log into regular account)

https://www.interactivebrokers.com/sso/Login

Left side of screen – "Trading Access" – under that – "Paper Trading

Account Reset"

It doesn't kick in until next trading day.

by Jim Covington

[Q] How API version maps to client/server version I see in a log?

[A] by IB support (by phone)

Client side:

API 9.67 —> Client version 59 TwsApiC++.9.67_3.zip

API 9.70 Release Date: October 2013 —> Client version 62

API 9.71 Release Date: Aug 6 2014 —> Client version 63

Server side:

build 946 —maps-to—> TWS socket server version 71

[Q] Client can not re-connect to TWS using the same client ID after connection was not properly closed.

[A] by Dmitry

https://groups.io/g/twsapi/topic/4047690

And followed discussion in the hijacked thread 😉

Same topic was also discussed here:

https://groups.io/g/twsapi/message/37644

Basically it is about: when client closes connection to TWS abruptly then client often can not reconnect using same client id. One would need to restart TWS or wait for things to self-release on TWS app side (up to about 1 hour).

[Q] How to change ib primary HOST? (moving to new region)

Just got a box in NY4 but my IB account still connects to ASIA – hdc1.ibllc.com even though I choose region as – America at login window…

Any config I can change to force IB connects to USA East – ndc1.ibllc.com?

[A] by Evgeni Andreyev e.andreyev07@gmail.com View/Reply Online (#45559)

Added on 03-Oct-2020

No, you have to explicitly request (from IB Support) moving the IB server assigned to your account at registration to a new location. This will then happen over the weekend.

[Q] TWS SLOW connection times?

[A] by JR <TwsApiOnGroupsIo@reinold.org> View/Reply Online (#46681)

Added on 03-Mar-2021

Since the tally of folks that have the same issue seems to be approximately zero, IB having an issue feels unlikely, though not impossible. So let me briefly raise three things that I'd look at if I'd have to troubleshoot API connection issues.

[1] Stale API connections:

We once wrote a simple "Is my TWS alive and actually connected to IB" monitor that would connect to TWS, do a simple operation that requires TWS to communicate with IB, and then disconnect. The monitor was started automatically and frequently but somehow caused stale (not completely closed) API connections to build up. When you click "DATA" to get the connections listing in TWS, do you see stale API connections from past script executions? Those did cause connection request delays, in my case, even when TWS was far away from the maximum number of API connections.

[2] DNS reverse lookup:

You said you checked the network on your server, but did you check DNS as well?

Have you experienced any sluggishness (even briefly) for regular commands on your server? Assuming it's a Linux system, even commands such as "ls -l"
Does "netstat -a" or "lsof" hang for a few seconds but "netstat -an" or "lsof -n" do not?

Several library functions unexpectedly perform "reverse DNS lookups" and can block the caller (in this case the thread in the Java JVM that handles your incoming API request to TWS) for extended periods of time until failing DNS lookups succeed or time out. If your internet service provider has any instability with reverse DNS lookups, otherwise perfectly fine servers can have operations seemingly randomly freeze for 20 to 30 seconds.

[3] How are your entropy levels?

This is a little stretch but fits the symptoms you are having if you are running some kind of Linux system. And it took me weeks to figure that out on a server I had in a data center in Dallas (though this is not specific to Dallas or Texas)

In order for strong SSL encryption to work, the Linux kernel collects "truly random events" and builds up a buffer of entropy that Java JVMs (and other programs such as sshd, httpd, …) access through the /dev/urandom device. And while only very few bytes are needed each time, opening or reading from /dev/urandom when the buffer is low or empty will block the caller until enough entropy has been collected. And the Java JVM frequently opens /dev/urandom when various network related operations take place.

You can read up on this with "man urandom" and check your server's entropy level with "cat /proc/sys/kernel/random/entropy_avail"

Hope this helps.

Code Examples

[Q] I'm total noob, where to begin?

[A] by ExStock View/Reply Online (#45602)

Added on 10-Oct-2020

I also started from scratch, so I don't mean to discourage you, because it can be done! But there is a lot that you'll probably need to learn in order to do what you want safely. (Be sure to spend a lot of time working with an IB paper trading account before trying anything out for real!)

One place you might want to start are the videos about the Python TWS API available from IB: https://tradersacademy.online/category/trading-courses/ibkr-api . These videos are a bit like drinking from a firehose–lots of information very fast–but they do cover a lot of the basics, and you can just hit pause a lot.

With those videos as background knowledge (and they do show the actual Python code involved for each example) you might want to consider installing the ib_insync Python framework, which simplifies the process: https://ib-insync.readthedocs.io/index.html . If you click on the Notebooks link, you can see more code examples.

You might also want to install either Jupyter Notebook or JupyterLab (you don't need both): https://jupyter.org/ . They both make coding in Python simpler, as well, and will make what you see in the ib_insync notebooks easier to follow and experiment with.

Finally, if you do get into ib_insync, there's also a groups.io for them: https://groups.io/g/insync .

All of the above is free to play around with–good luck!

[Q] Is there a book you may recommend for a newbie like me, to really know what I am doing, what I should look for doing and what I should carefully understand and avoid?

[A] View/Reply Online (#45571)

Added on 06-Oct-2020

Algorithmic Trading with Interactive Brokers by Matthew Scarpino has some basic python scripts and a few list of parameters that is useful for a beginner.

That book is good, I'd also recommend looking at Rob Carver's blog (https://qoppac.blogspot.com/) he's got a section (March 2017) on how to use the IB API, it covers the basic stuff connecting, getting prices, trading, reconciling which you can then build on.

[Q] Where can I find C++ and C# tutorials/examples using ActiveX?

[A] by Rich Holowczak (a.k.a. “Prof. H.”) (src)

Folks

A while back a wrote a series of tutorials for the Interactive Brokers API for my Financial IT classes. I am now getting around to assembling them on my web site. I have a few going up now using Visual C++ and Windows Forms. I will be posting more over the next few weeks that include WinForms with C# as well. Your comments and suggestions are welcome:

http://holowczak.com/category/ib/

Cheers,

Rich H.

[Q] Where can I find working posix C++ code examples?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A1] by Jim Covington

Sample code is on your machine inside the IB API installation folder.

[Q] Where can I find MORE ADVANCED working posix C++ examples?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] There is a great IB API Wrapper TwsApiC++ by Jan Boonen (see on GitHub)

(quote)

In the C++ implementation, the basic two API classes are EWrapper and EClient, in combination with the other classes/structures defined in the directory 'Shared'. Comparing with Java, these classes are pure interfaces. The implementation of EClient is done in EClientSocketBaseImpl and at linking time, all is connected. The implementation of EWrapper is up to you.

Almost every C++ programmer opts to write the definition of a class and its implementation in two distinct files, although C++ does not enforce it. I prefer more the Java way, although C++ does not support it to the same extend. Some care is needed, and when pure interfaces are needed, the definition and implementation has to be separated anyway. So when studying different C++ projects, you will discover different approaches. Discussing all the pro's and contra's is a never ending story, as are the programming text style discussions.

The non-posix version of the api cannot be used as stand alone. It depends on the Windows specific socket implementations for reading the socket and calling the EWrapper methods. So the 'EReader' is somewhere down deep in the windows specific asynchronous socket stack.

This makes the non posix version non-portable and unusable for a non-windows program. To overcome these two weaknesses, some free libraries exist. I started the free TwsApiC++ library in 2008. The current version is based on the posix library of IB.

What does this TwsApiC++ library offers you:

a) the EClient and EWrapper are still the basic two API classes (or more specific, a derived version of it EClientL0 and EWrapperL0). So no need to access all the internal classes such as EClientSocketBase, or the other complex posix classes.

b) it implements an EReader, which is turned on by default. It runs in a separate thread, and waits for the incoming data on the socket, and calls the EWrapper methods without any delay. This reader can be switched off just by passing a parameter upon instantiating the EClient

c) it implements an empty method for every EWrapper method defined. So you only have to implement the one you are interested in

d) It protects the inner working of the IB library, where the TwsApiC++ is a wrapper around, from the possible exceptions thrown in the users code, but not catched by it.

e) It protects the EClientSocketBaseImpl from concurrent use of its internal members, which could lead to loss of requests send to the TWS

f) It ensures the data stuck in the buffer of EClientSocketBaseImpl and failed to be send to TWS for some reason, is send as soon as possible, and not sits there waiting until the next EClient call is made. In the posix api, you the programmer has to take care of that(!)

g) optionally, it provides a way to check the correct spelling of the many textual or numeric parameters upon compile time instead of runtime.

The library is small and fast. A few examples/test programs are included, and also project files for Visual Studio and make files for linux/unix are included

It includes a readme file.

See it on GitHub:

https://github.com/JanBoonen/TwsApiCpp

Or for group members:

See Files > C++ Code > TwsApiC++ Directory

There are other implementations too, other members of this group know more about these.

Have fun with it.

Jan

(end of quote)

see also tutorial how to install TwsApiC++ here: [Q] Installing IB Gateway and TWS API for linux? (tutorial by Rich Holowczak (a.k.a. “Prof. H.”))

Note from Kurt:

when I *was* using TwsApiC++ I had to do extra work to *undo* the fact

that (like Java, I guess) TwsApiC++ creates a separate thread to read the

socket, and callback messages come in on that "reader thread". So I needed

another layer of buffering to get the callbacks to come in on the main

thread, and this required a tricky set of function overloads which pretty

much required modifying the TwsApiC++ implementation classes due to how

things were (not) factored.

The posix TWSAPI either does not require any such effort. Either there is

not a separate reader thread, or the buffering is already taken care of. (I

can't remember which.)

-Kurt

Another note from Jan came from here

JanB

Message 16 of 28 , Oct 20, 2010

View Source

Hi all,

I second this: the new improved API shouldn't reinvent the wheel, but rather make the current IB API a 'more round wheel and easier to turn' (hope this makes sense).

As Shane Cusson states, implementing a simple strategy with IB's API is 'like using a bazooka to swat flies'. So this new API should simplify its use, but should not prevent the implementation of complex strategies either.

For the C++ implementation, something like TwsApiC++ is a first step as it hides the use of the sockets. Not ESocketClient is the interface, but EClient/EWrapper is. Having to deal with the sockets in your ATS is fundamentally wrong, isn't it? TwsApiC++ more or less levels the C++ API with something like the API from dinosaurtech for C#.

On top of that, I'm implemented/ing two extra layers and I thought it could be interesting to share them here:

In a first layer on top of EClient/EWrapper (EClientL0/EWrapperL0):

1) The functionality of EClient/EWrapper is regrouped in functional classes such as Connection, MktData, HistoricalData, PlaceOrder, Account, Portfolio, etc. Each class keeps the data needed to make the request, and all EWrapper events related to the request are routed back to the requesting class. I.e. each reqHistoricalData to be executed needs its own instance of the class HistoricalData, which handles the request and receives all the historicalData() events for only his request.

2) It hides all the Id's (tickerId, orderId, reqId ): the allocation of the id's is automated and hidden. Errors with id==-1 are routed to the Connection class.

3) The classes for Account and Portfolio filters the events on the acctCode, so each account can have its own instance of Account and Portfolio. A blank/empty acctCode makes it to receive the data for all the accounts in case of.

4) Each of these classes have a common interface of a Request(), Cancel(), Resend() and Error(). The parameters for the req… are set during the allocation of the classes, so the Request() method has no parameters.

As the first layer resembles the EWrapper interface and does not adapt to any special behaviour of the IB API, the second layer goes one step further to make live easier for the programmer as follows:

1) it stores the data for classes such as Account, Portfolio, HistoricalData etc. (not for Mktdata or RealTimeData i.e. )

2) offers a more uniform interface when possible: OnInit() (called after the …End() events meaning all Account is available, Historicaldata is complete etc), OnChange()(i.e. after the Init() was called, and new updated Account data receives), and a ForEach(), forward and backward Iterators, etc. to access this data.

3) Split up of the MktData into MktDataSTK, MktDataFUT etc. The work-around for the "zero size bug" of Mike Smith does fit in this layer nicely.

4) a connection persistence and Resends the requests in case the link was broken with error (1101), 1102, 1300

5) A run method in the Connection class: i.e. Run(Run::Until(15,30,00)); which basically loops and calls checkMessages() and Connection::RunActions(). So my ATS or test programs have all a very compact 'main' routine: allocate the ATS and a call to the Run method.

I hope this view, although implemented in C++, is of any help. The implementation is not complete yet.

Jan Boonen

[Q] Python?

[A] For IB API + python awesomeness please check out:

Free udemy course: "Python programming with Interactive Brokers' Trader Workstation API"

https://www.udemy.com/course/python-api-for-trading/

Also:

For the Python language, Ewald de Wit has created an amazing framework called ib_insync that takes care of all the housekeeping and management functions and let’s you focus on your client code and trading. In case you are not aware of it, take a look at his GitHub repository and the extensive documentation. ib_insync allows you to operate the TWS API in its natural asynchronous request/response mode but also provides convenient blocking versions for more traditional sequential approaches. You can also use it as a rich source of examples on how you can achieve synchronization in Python.

Just take these two lines of code as an example. This is all you need in ib_insync to connect your client to the TWS API

ib = IB()

ib.connect('127.0.0.1', 7497, clientId=1)

Compare that with your code where you had to spend about 75% with boiler plate and connection related activities.

The ib_insync package: https://github.com/erdewit/ib_insync

and related discussion group:

https://groups.io/g/insync

Also: sprach groupse.io forum by word “jupyter”

https://groups.io/g/twsapi/search?q=jupyter

And then also:

https://groups.io/g/twsapi/message/37515

Use a GUI library like tkinter or pygame. For testing you could also use jupyter, there's a sample in here somewhere.

https://groups.io/g/twsapi/message/37329

https://groups.io/g/twsapi/message/37341

[Q] Delphi?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by josejp1

In this link are good examples of programming of Trader Workstation API with Delphi

http://delphimagic.blogspot.com/2010/07/interactive-brokers-con-delphi-tws.html

Ej. Connect with TWS, request market data, get ticks of price, place an order (mkt, lmt, stp, etc), request historical data, get information of an order, how to cancel orders

— also —

Delphi / BCB / Kylix TIABSocket component & API

http://www.hhssoftware.com/iabsocketapi/index.html

updated to the latest 9.76 API (January 2020)

This component set is built to work with the Interactive Brokers TradeWorkStation API. It allows Delphi and BCB programmers to build apps that send orders directly to the TWS. This TIABSocket component links directly to the socket to communicate with the TWS. There are no dll, ActiveX, .Net or DDE files required, and all socket messages & data processes instantly. This direct connection to the socket makes for a fast reliable interaction and response time. The component set also controls all interactions with the TWS, and includes sub components to manage order activity, portfolio and account details. The code looks and feels like any other Delphi component, and the syntax is similiar to existing standard component code. The complete source is included, along with a demo app and help file.

[Q] Java?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by Suminda

I have started a CEP (Complex Event Processing) based trading system which will be agile as possible. This is at a conceptualisation stage where we are running a few experiments on how the final architecture will be. Is it possible to help out in this as well as help build the team to take this forward.

Suminda

http://code.google.com/p/cep-trader/

java sidenote: by hubert.tse@gmail.com

My Eclipse was set on Java Compiler -> 1.5 Compliance Level.

Apparently 1.5 does NOT support @override and thus will not run. Once you set this to 1.6 and beyond, all the errors go away and it launches perfectly.

Many thanks for all your help!

—

[A] by Rob Terpilowsky (thread)

The 2 libraries I've used are

ta4j: https://github.com/mdeverdelhan/ta4j

and TA-Lib :http://www.ta-lib.org/

I prefer ta4j as it has a cleaner API than TA-Lib.

The 2 libraries I've used are

ta4j: https://github.com/mdeverdelhan/ta4j

and TA-Lib :http://www.ta-lib.org/

I prefer ta4j as it has a cleaner API than TA-Lib.

[Q] ruby?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by Dr. Hartmut Bischoff

Well, there is ib-ruby, too.

You can use pure ruby scripts, like

require 'ib'

ib = IB::Connection.new :port => 7496

ib.subscribe(:Alert, :AccountValue) { |msg| puts msg.to_human }

ib.send_message :RequestAccountData

ib.wait_for :AccountDownloadEnd

ib.subscribe(:OpenOrder) { |msg| puts "Placed: #{msg.order}!" }

ib.subscribe(:ExecutionData) { |msg| puts "Filled: #{msg.execution}!" }

contract = IB::Contract.new :symbol => 'WFC', :exchange => 'NYSE',

:currency => 'USD', :sec_type => :stock

buy_order = IB::Order.new :total_quantity => 100, :limit_price => 21.00,

:action => :buy, :order_type => :limit

ib.place_order buy_order, contract

ib.wait_for :ExecutionData

https://github.com/ib-ruby/ib-ruby

—

Dr. Hartmut Bischoff

Stuttgart

Update 06-Jun-2018: by Dr. Hartmut Bischoff:

View/Reply Online (#39948)

(quote)

Imagine you got a trading robot at a remote location, accessible only via an reverse ssh tunnel, or a TWS is running on a cheap cloud-server.

Simple-Monitor ( https://github.com/ib-ruby/simple-monitor ) is designed to display the status of a Friends&Family-Account via web-interface.

It comes with just 435 lines of code using the camping micro-framework and ib-ruby.

Its intended to run even on resource-low hardware and serves well running elinks in a tmux-session, avoiding a running TWS-GUI in favor of the TWS-Gateway

It too demonstrates how well ib-ruby ( https://github.com/ib-ruby/ib-ruby ) integrates in existing frameworks.

(end of quote)

[A2] by Peter Gum

IBCsharp, tradecommander, openquant,

tradelink and others. RighEdge <http://www.rightedgesystems.com/> is as

comprehensive as any, including supporting VB.NET

google "jTWScharts"

[Q] C#?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A1]

see: Interactive Brokers C# Api (by Karl Schulze)

ib-csharp

https://github.com/krs43/ib-csharp

and/or:

http://www.dinosaurtech.com/utilities/

[A2] by Rich Holowczak (a.k.a. “Prof. H.”)

I have written a few more to work with IB's C# API (console only applications). same location:

http://holowczak.com/category/ib/

Cheers

Rich H.

if you aware of any other published working code examples, please let me know so I can add the link into this FAQ

[Q] R

[A] by ce (from this thread)

pdf files in

http://cran.r-project.org/web/packages/IBrokers/index.html

are good.

Also there is a very good example system in

http://censix.com/download/

[A2] by censixtws (from this thread)

Hi I have developed this collection of R scripts that I call qsiblive ( downloadable from my website http://censix.com ) and am currently testing it with my paper account. It trades one symbol on an EOD basis. I think its strength lies in the immense flexibility that it provides with regard to 'plugging-in' R analytics packages/methods (some classic indicators, some ARMA, GARCH modeling, you name it). Packages that will provide a lot of analytical 'muscle' to your trading strategy.

I am making this available to get some feedback from the community of seasoned traders/developers here.

Cheers

Soren

see:

censix – pure R intraday trading framework – screencast01

https://www.youtube.com/watch?v=Zzd7tx452Ao&index=2&list=LLYax4ZgrNYHP5fLotKBc9Lg

censix on GitHub: https://github.com/censix/INTRADAY-PartAB

[R-related resources]

Contemporary Portfolio Optimization Modeling with R

https://www.youtube.com/watch?v=hKDalfhDawA

[Q] Pattern recognition library?

Added on 16-May-2017

is there an open source library for chart patterns? or at least algorithm so i can use to program it myself?

looking for flag, triangles and double top/bottom mostly.

[A] by Johnny Red (View/Reply Online (#37504))

I use TA-Lib – it's open source and very good. It's written in C++ and is available for Java/C#/etc. I even think there is a version for Python.

Good stuff.

http://ta-lib.org/hdr_dw.html

[A2] by Peter Cawthron

There’s a Python wrapper for TA-Lib at https://github.com/mrjbq7/ta-lib

Regards,

Peter

[Q] Reverse engineering java?

[A]

I know you mentioned you've resolved your issue, but just FYI, if you want

to peek at the contents of "other.jar":

http://java.decompiler.free.fr/

Grab the GUI version and click "File->Open", pick "other.jar" and browse the

source. The functionality is essentially the same as the .Net Reflector.

–s

ps. if reverse engineering is illegal in your country, you didn't hear this

from me. (I'm Canadian, things are pretty lax up here.)

——- also ——–

Alternatively, make a copy of it and rename to "other.zip" and open as

usual with Windows or your Mac.

Implementation notes

[Q] How do I retrieve the bid/ask spread for given contract?

[A] View/Reply Online (#46694)

Added on 13-Mar-2021

I was working on it recently, so let me try to put a 1st draft of the answer:

[Q] How do I retrieve the bid/ask for an option leg?

[A] I found 3 ways so far. Only one works for me, but I'll mention all 3 here.

1-of-3: Using client.reqTickByTickData(19003, ContractSamples.USStockAtSmart(), "BidAsk", 0, true);

Read this article: TWS API v9.72+: Tick-by-Tick Data

In particular read about IBApi::EWrapper::tickByTickBidAsk, which would give you bid/ask.

The problem here is that you only allowed only 3 simultaneous subscriptions, which might work for someone who's training 1-2 instruments, but I wanted to find more generic way to get bid/ask for more contracts at the same time.

2-of-3: Using EClient.reqMarketDepth (reqMktDepth in Java, C++, and Python). See these 2 docs:

TWS API v9.72+: Market Depth (Level II)

TWS API v9.72+: EClient Class Reference

In their docs IB says:

Beginning in API v974 and TWS v974 it is possible to Smart-route a IBApi.EClient.reqMarketDepth request to receive aggregated data from all available exchanges, similar to the TWS BookTrader display.

(end of quote)

And I come to the conclusion that this "aggregated data" is the only right way to get proper bid/ask unless you're always trading some contract that is only traded on 1 particular exchange and thus you don't need to aggregate data from many exchanges. If you use "reqMarketDepth" with contract.Exchange set to SMART and set "isSmartDepth" argument to true, then you'll get an error, that you require more subscriptions. In my data subscriptions I didn't have any Level-II except a free one for "IEX exchange". If I try to set contract.Exchange = "IEX" and set "isSmartDepth" argument to false, then I start getting data, but noticed, for example for "IBM" contract, that the bid/ask during any quiet low volatility day (shown in TWS around $0.02), is reported by IB API often times as a much bigger number (for example $2.45 vs $0.02 expected – see screenshot below). IB support confirmed that what you see in TWS bid/ask (with list of different exchanges) is NOT accessible via API without subscription, but is accessible for your "human naked eye" for free in TWS window (it's time to engage OCR!-)).

Here's an example when IB API returned $2.45, while TWS shows aggregated data with bid/ask = $0.02 around Feb 8, 2021, 2:47 PM

You can see IEX Bid is 121.0 and IEX Ask is 123.45, so the spread by "IEX only" version is 2.45! Nasty and useless!

If you want to use SMART for the "reqTickByTickData" you'll have to have Level-II subscriptions for ALL exchanges this contract is traded on.

To check how much all of them will be login to interactivebrokers.com -> AccountManagement -> Settings -> User Settings -> Market Data Subscriptions ->

You'll see "North America: IEX Depth of Book – Trader Workstation – Fee Waived"

Along with notes:

IEX exchange depth of book service. Requires prior subscriptions to NYSE (Network A/CTA), AMEX (Network B/CTA), and NASDAQ (Network C/UTP) or US Equity and Options Add-On Streaming Bundle.

(end of quote)

Then click: Current GFIS Subscriptions -> configure gear icon -> North America (as example) -> Level II (Deep Book)

And you'll see the list with prices.

3-of-3: poor-man bid-ask using real-time bars and "ticks"

If you subscribe to two real-time 5 seconds bars streams by either:

a) IBApi.EClient.reqHistoricalData request with "keepUpToDate" set to true (see also TWS API v9.72+: Historical Bar Data)

b) IBApi.EClient.reqRealTimeBars request (see also TWS API v9.72+: Historical Bar Data)

And specify "what to show" to "BID" for one request and "ASK" for another request, then IB API will call back with 5-second bar updates every few seconds (on average every 3.5 seconds) and each stream "bar.close" price will represent most recent bid and ask. The only problem is that those 2 streams represent 2 events streams (separated in time). I filter out negative calculated bid_ask spreads as well as calculate bid_ask spread only if bid update and ask update are within 500ms distance. As a result on the same "IBM" ticker I got on average (during the whole trading day) calculated bid_ask values coming on average every 3.5 seconds, which is good enough for me 🙂

Note #1: this method is only counted as "one line" out of your 100 available lines, so theoretically you can get 100 bid_ask spreads (along with sampled "real time tick" for price updates) without buying any data-boosters or Level-II subscriptions.

Note #2: yes, there is "what to show" type "BID_ASK" (see Historical Data Types), but it will not give you an instant snapshot of the current most latest bid and ask values. Instead you'll get:

bar.open = "Time average bid", bar.high = "Max Ask" (useless), bar.low = "Min Bid" (useless) and bar.close = "Time average ask". I'm not sure if the "time average" values for 5-sec bars will be better than the above method, I'll try to check them side-by-side, but my guess is – both approaches will yield similar quality output.

I'm not sure how well this method would cope with fast moving markets. I guess it will give quite a random estimate for what really is. But one can always use some safety factor (for example use calculated bid_ask spread value and multiply it by 2.0 just to be sure). Also it would be interesting to see how the spread value correlates with the current volatility. During volatile times the price might jump, the bid/ask might jump, but the high volatility reading will be showing pretty much flat high value, which you can use to estimate the bid/ask max value (and/or to adjust the "safety factor" and make it a bit higher when market goes crazy:).

There's some free 15-minutes delayed bid_ask stream if I recall correctly (as 4th option), but I haven't looked into that option since the 15 minutes delay is a bit too far.

Cheers,

Dmitry Shevkoplyas

ps: I'll show here the function I used on every incoming bar to prototype the bid_ask calculations, think of this piece as "meta language":

public void on_historical_bar_try_to_calculate_bid_ask_spread(

Bar bar,

RequestHistoricalData request) {

// Check we have both requests in place (not nulls)

if (request_historical_data_bid == null) {

logger.debug("request_historical_data_bid == null, nothing to do yet…");

return;

}

if (request_historical_data_ask == null) {

logger.debug("request_historical_data_ask == null, nothing to do yet…");

return;

}

// Both bid / ask "real time" bars requests are present! We can calculate bid/ask spread estimate.

// Let's extract most recent bars from 2 streams:

Bar bid_bar = request_historical_data_bid.most_recent_bar_received;

Bar ask_bar = request_historical_data_ask.most_recent_bar_received;

if (bid_bar == null) {

logger.debug("bid_bar == null, nothing to do yet…");

return;

}

if (ask_bar == null) {

logger.debug("ask_bar == null, nothing to do yet…");

return;

}

// Check how far in time they are from each other

long bars_received_time_delta_ms = request_historical_data_bid.most_recent_bar_received_time.get_delta_ms(request_historical_data_ask.most_recent_bar_received_time);

// Filter out updates with more than 500ms distance

if (Math.abs(bars_received_time_delta_ms) > 500) {

return;

}

// Both bid/ask bars are present and close enough, let's calculate the bid/ask spread

double bid_ask_spread = ask_bar.close() – bid_bar.close();

// Filter out some obvious BS

if (bid_ask_spread <= 0) {

return;

}

logger.info("Calculated synthetic poor man's bid_ask_spread: "

+ Aid.math_round(bid_ask_spread, 2)

+ " bars_received_time_delta_ms: " + bars_received_time_delta_ms);

}

[Q] How to close an OCA group?

[A] View/Reply Online

Added on 22-Feb-2021

Nick: You can also modify the stop order to be a market order. This may not work with all order types so you have to try it with your scenario.

[nice and elegant solution!]

ds-avatar: TWS will automatically assign an OCA (one-cancels-all) group id to the child stop loss order. That's how it implements the execution of only a single child order if more than one are submitted. You can use it for your manual market exit order so its execution will cancel the child order.

[Q] Order field to identify a strategy? (clientID vs orderRef vs permID)

I have multiple strategies running on the same account/sub-account.

I want to distinguish which orders were placed by who.

Is there a field in the Order class that I can assign a custom value, and that value to be persistent so I later can identify who placed the order?

Someone suggested to use orderRef field.

Similar question: by Corneliu Maftuleac corneliu.maftuleac@gmail.com View/Reply Online (#46716)

I am using the orderRef field of an order to mark an order as it belongs to a specific strategy.

However when using flex queries to retrieve the list of orders the OrderReference field is always empty.

As specified on the docs: The order reference number as defined by the user on the order ticket.

Any ideas how to retrieve order reference?

[A] View/Reply Online

Added on 22-Feb-2021

There is OrderRef but it might not persist across sessions, you would have to make sure it meets your needs.

By "Session" Nick meant:

Except for forex, markets are open for only part of a day as opposed to continuous 24/7 trading.

The general idea is markets open at a specific time and then close at a specific time and that is one trading session. Trading on the next day is a different session.

Different financial instruments have different rules for when a session starts and ends. A single session on the major futures exchanges spans calendar days – opening in the evening one day, going past midnight until the afternoon or the next day. Can make you crazy since a trade can occur on Monday evening but counts as the Tuesday trading day.

Back to the OrderRef, people have observed that the field is returned for executions in the same session as the order was placed but is empty if the order executed in a subsequent session.

Mark Colling also suggested OrderRef:

I use trade references stored in the order reference field, it usually works without problem. Track the order id, order reference and your internal tracking identifier, and you should be able to piece it all together.

What I have experienced is position reports being lagged by many minutes, and execution reports arrive in an arbitrary order, but the order references do seem to be stable.

Answer by ds-avatar:

OrderRef will be in reports only for the most recent session. Workaround: configure the system to send daily reports by mail, then collect and collate them.

[best answer by praditik]

I have used permID and stored it locally to identify strategy mapping. It works seamlessly across sessions and days. So as everyone else already said, I think, using permID with local mapping in your program should be able to do the trick.

Resolution:

Yes, the OrderRef is unfortunately eventually gone, but the report should also carry order.permID field, which is account-unique identifier, and it should be there forever.

So you can keep track locally of all your ordersRef values and corresponding permID assigned to your order. Thus many months after, when you pull the report from IB you'll be able to restore original orderRef value by lookup into your local table (orderRef <–> permId). I agree – the IB's report should be self-sufficient and I don't understand the reasoning of deleting some of the existing fields (if it is there – it should be there forever), but it is what it is. Please let me know if it works for you? Do you see permID in the report?

Corneliu Maftuleac: – Yes, we see permID in the report.

[Q] Detection of loss of connectivity

[A] View/Reply Online

Added on 04-Feb-2021

Eugene,

Yes, there was very similar topic discussed around September 2015 in this thread (subject: asynchronous EClient::eConnect() needed?) and the conclusion was that if you just connected to TWS API and don't send anything then you might get hard time detecting it quickly (and in some cases you'd even get stale connection on the TWS side and won't be able to re-connect to TWS using same client ID without restarting it (or after something like 30-40minutes)).

The solution to this would be to subscribe to something "noisy enough" so TWS site would try to use established tcp connection to your client app as well as in parallel keep asking from your client to TWS API something simple, like reqCurrentTime() every second (TWS answers it on its own, without bothering/asking remote IB's servers, so it is safe to be annoying:) Those 2 "streams" will be the "heartbeats" you're looking for!

hope it helps,

Dmitry

Hi Kurt,

Seems in my 1:30am reply I slipped to the "client <==> TWS" detection instead of "TWS <==> IB".

As Eugene mentioned api client only detects connection lost when it tries to send order (or perform other api call to TWS), which is exactly the problem I observed about 1yr ago in mentioned above thread when my client was running on other than TWS box. So to mitigate it (and make api client aware about its connection lost to TWS quicker) client should try to talk to TWS. Also TWS clears its internal connection-describing tables much quicker if it fails to deliver something to client hence suggestion to tickle both directions by asking time (client -> TWS) and subscribe to some ticker (TWS -> client).

As for "TWS <==> IB connection lost" while "client <==> TWS" connection is fine (say for simplicity api client and TWS are on the same host) then TWS indeed have some mechanism (like heartbeats) – even when you aren't subscribed to anything in TWS you can see every 10sec remote IB host (in my case cdc1.ibllc.com, which also can be found in ~/Jts/jts.ini) pushes packets with content like this (truncated packets payload): "FIX.4.1.9=00014.35=1.112=farm.10=217" and

"FIX.4.1.9=00027.35=0.112=FixTestRequest195.10"

and then TWS reports back to IB something FIX-protocol-related, like:

"FIXCOMP.9=70.x"

And when "TWS <==> IB connection" is lost then api client is notified by TWS immediately. So luckily for us there's no need to invent a new magic wand to cause TWS to "share" with the client app information previously unshared about its awareness of its connection with IB – this logic is already there. My TWS reports connection lost within 30sec and recovers within 5-10sec upon network restored.

Sorry for not being clear with my midnight statement 🙂

thanks,

Dmitry

[Q] Get list of all symbols by exchange?

[A] View/Reply Online (#46323)

Added on 29-Jan-2021

NASDAQ has API, which gives you list of all symbols and even their last price as JSON:

https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=25&offset=0&download=true

[Q] Getting stocks / options standard deviation?

Is there a way in TWS to get a simple option analysis graph?

Do you know how to calculate the stock price standard deviation?

The goal is to get via API all necessary data to calculate this SD for any stock/options.

[A] View/Reply Online (#44329)

Added on 03-Oct-2020

Take a look at HISTORICAL_VOLATILITY on reqHistoricalData.

https://interactivebrokers.github.io/tws-api/historical_bars.html#hd_what_to_show

— also

In TWS, you can right clight on an option and ask for its performance profile, it's almost the same window.

If you use reqMktData on a stock contract with IDs 104 and 106 you will get annualized HV and IV for a specific stock

You can fetch annualized IV for a specific option contract with the ID 106, but not the HV

You could also compute the HV by yourself if you have more than 30 ticks, as I believe IB uses Garman Klass formula with a window of 30. But you can use any other volatility formula I suppose.

Of course to get an accurate range for a given date, you would need to adjust the volatility to the period of time. If you want the volatility over 15 open days: VOL / ((15/252) ** 0.5)

Hope this helps

[Q] How to detect TWS disconnected from API?

On Wed, May 20, 2020 at 02:45 PM, Spread Haendler wrote:

I am using TWS 978.1s and latest TWS API 9.79.01 (hopefully latest API version).

If the TWS disconnects from API (unfortunately I have frequent sudden shutdowns of TWS978), I receive no error message via API.

[A] by rwk, View/Reply Online (#44515)

Added on 04-Aug-2020

It seems risky to depend on a disconnect message to signal that something has gone wrong. We could have a good connection and still not get quotes or be able to place orders. What I do is monitor a contract such as an index ETF that I know should be active. After some interval, such as 20 seconds without data, I raise an audible alarm and investigate.

[Q] What value means "no value"?

[A] by Kurt (src)

IB shows no consistency in the use of values meaning "no value". Sometimes it is INT_MAX or DOUBLE_MAX, sometimes 0, sometimes –1, sometimes –0.99, etc.

The historical bar volume field is only populated for TRADES, never for BID, ASK, MIDPOINT, BID_ASK, OPTION_IMPLIED_VOLATILITY, or HISTORICAL_VOLATILITY. Instruments that do not provide TRADES (such as FX) therefore do not have volume available.

There are probably many other things omitted for various instruments in ways that follow no particular pattern, and are not documented. You just get what you get. Don't worry about things looking strange. That is the least of the problems you will have at IB.

I've definitely noticed that in historical data the unused fields in a bar are sometimes 0 and sometimes –1. There is definitely no documentation about that.

–Kurt

[Q] Best practice to handle ID’s / nextValidId?

[A] by edbar@exmsft. from this thread

I store the ID on my hard drive.

Then when I start up TWS, I check the nextID that they send to see which one

is bigger.

If my stored ID is greater than the TWS number then I go with mine.

Then, every time I increment the ID number, I update my hard drive backup.

You can always go to the TWS CONFIG API screen and reset the api id numbers.

–Ed

[A2] by btw12342001 (same thread)

I don't think you should use the orderId for tickerId. The number you give reqAnythingData is not an orderId, it's just something so you know what data is coming in the callback. It's possible you subscribe to more than one symbol of data. It still doesn't explain why you're mixed up with orderId's

I guess I don't need to answer the 2002 question 🙂

[A3] by Kurt (same thread)

A lot of people have reported here that just using nextValidId() is not

sufficient. I've been doing it without problems so I may not recall the

details, but I believe the common practice it to keep track (on disk) of

your greatest order id, and use the greater of that and the value returned

by nextValidId when you start up, and increment from there. You can check

the archives. This question was last asked less than 2 months ago I

believe.

Regarding reqMktData etc each one has its own id space but for your own

sanity you probably want to keep them unique. I keep all request id's

unique *except* for placeOrder for which I used a separate scheme. Using

unique request ids facilitates routing incoming request errors.

-Kurt

[A4] by Richard L King rlking@aultan.com View/Reply Online (#44680)

Added on 10-Oct-2020

There is no requirement imposed by the API to use different ranges of ids for the various requests that use them (about the only rule is that order ids must never decrease, except that you can reset the order ids for a particular clientid via TWS). Moreover, there’s nothing to prevent you using the same id for a market data request, a historical data request, an order etc all at the same time.

As you’ve observed, this causes potential problems with the error callback.

One way to address this is to define your own ranges. For example, 0-1023 for market data requests, 1024-2047 for contract details requests, and so on. You of course have to think about what happens when you get to the end of a range, and there are many ways of dealing with this, and such considerations will guide you in deciding how big the ranges should be. For example, contract details requests complete fairly quickly, so you could almost certainly get away with a range of say 1000 ids for them, and just wrap back round to the beginning when you get to the end – the chances of the first request not having completed by the time you’ve made the next 999 requests is remote: however I personally wouldn’t take the risk, and I’d use some kind of data structure to keep track of ids which had been used and those which were free.

If you do this, then it’s trivial to decide, for any error message that includes a non-negative id, which type of request it relates to, and even which precise request if, for example, you use the ids to index into a table per request type.

I use such a scheme. In particular I use 0x10000000 to 0x7FFFFFFF for order ids. This large range means I’m not going to run out of order ids in my lifetime (there are 1,879,048,191 of them!), so I’ll never have to use the order id reset mechanism. I also ensure that within this range, ids for single orders and entry orders in brackets end in 0 or 5, stop-loss orders in brackets end in 1 or 6, and take-profit orders in brackets end in 2 or 7: this is simply to make it easier to identify the role of an order when looking at a logfile.

A scheme of this sort should be entirely compatible with your use of CompletableFutures.

Richard

[Q] After connection loss I need a list of all open orders and positions in order to get synchronized?

[A] by src

Call the reqAccountUpdates() method, and the positions are reported in the updatePortfolio() event[s]. The events fire once for each position. Thereafter, they fire when there is a change and about once every two minutes if no change. It works well.

[rwk]

You might consider also requesting executions. Sometimes open orders or

execution reports (or both) may fail to work right, due to IB misfeatures or

whatever. The more information the better, so that means requesting

positions is also a good idea. Positions are part of account updates.

-Kurt

> What do I send in the acctCode parameter of reqAccountUpdates()?

You can set it to the null string if you do not have an FA account scenario.

-Kurt

or you can set it to: your account code UXXXXXX or FXXXXXX

> What do I receive in the position parameter of updatePortfolio()? Is it the

number of contracts in the Portfolio

YES

[Q] Tracking time (internally)

[A] by Dmitry

This has nothing to do with any incoming timestamps from outside – it is just for local timestamping of all internal events. As the good enough resolution I choose millisecond resolution. Local time is kept my NTP with updates polled from ntp servers as often as we can (had to build own Stratum-1 ntp server based on gps pps signal (see Time drift for more details))

long get_epoch_ms()

{

// http://brian.pontarelli.com/2009/01/05/getting-the-current-system-time-in-milliseconds-with-c/

timeval t;

gettimeofday(&t, NULL);

long epoch_ms = (t.tv_sec * 1000) + (t.tv_usec / 1000);

//std::cout << ";" << t.tv_sec << ";" << t.tv_usec << " or simply epoch_ms: " << epoch_ms << std::endl;

return epoch_ms;

}

[Q] Building live Trades

see also relevant: [Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] What is the sequence of tickSize() tickPrice events?

[A] by Dmitry

If you want to extract events for closed deals (not some BID/ASK moves noise, but really closed deals with real price/volume), then you gotta follow these considerations since it might be not quite straight forward due to IB would only send relative “differences” instead of absolute self-contained discrete messages about trades. To warm up, please read: [Q] Sometimes lastSize=0. wtf?

Now the implementation algo to get trades Dmitry uses:

– have container for trades within MyContract class

– all tickPrice(), tickSize() events are routed by ewrapper to request object (RequestMktData), which is associated with MyContract class instance, which in turn has tickPrice() tickSize() methods to process incoming ticks

– now inside MyContract class:

void MyContract::tickPrice(Account *account, long epoch_ms, TickerId tickerId, TickType tickType, double price, int canAutoExecute) {

LOG(INFO) << "MyContract::tickPrice";

// forward raw tick to subscribed algos (if any)

for (std::vector<AlgoBase*>::iterator ialgo = subscribed_algos.begin(); ialgo != subscribed_algos.end(); ialgo++) {

(*ialgo)->tickPrice(account, this, epoch_ms, tickerId, tickType, price);

}

// analyze if GotTrade()

if (tickType == LAST) {

// we got new trade, but we have to wait for TickSize() call to

// complete the Trade obj (and then will call GotTrade(trade) to pass

// complete Trade object to all subscribers)

this->tradesPerAccountMap[account_as_key].push_back(new Trade(account, epoch_ms, tickerId, price));

}

. . .

}

void MyContract::tickSize(Account *account, long epoch_ms, TickerId tickerId, TickType tickType, int size) {

LOG(INFO) << "MyContract::tickPrice";

// forward raw tick to all subscribers

for (std::vector<AlgoBase*>::iterator ialgo = subscribed_algos.begin(); ialgo != subscribed_algos.end(); ialgo++) {

(*ialgo)->tickSize(account, this, epoch_ms, tickerId, tickType, size); // forward tickSize() event to subscribed algos (if any)

}

// analyze if GotTrade()

if (tickType == LAST_SIZE) {

. . .

// CASE 1 of 3: check if last_trade is in "waiting for size" state (last_trade->size < 0 aka 'uninitialized'))

if (last_trade->size < 0) {

// last_trade's size is not yet initialized, means we waited for this tickSize() event

// to complete the trade obj (and forward it to all subscribers)

last_trade->size = size;

last_trade->completion_time_ms = epoch_ms – last_trade->epoch_ms; // we track how long it took us to complete this trade object

last_trade->is_complete = true;

// forward complete trade to all subscribers

gotTrade(last_trade);

return;

}

// CASE 2 of 3: Is it duplicate of last_size tick (same size as last complete trade and within 9ms) => just ignore it

if (last_trade->expect_dup_ticksize && last_trade->size == size && (epoch_ms – last_trade->epoch_ms) <= 9) {

last_trade->expect_dup_ticksize = false; // mark last_trade as "got expected duplicate" so next tickSize if

// happen to arrive within <7-8ms with the same size field would not be treated as duplicate anymore

// (the 'expect_dup_ticksize' flag allows us to ignore tickSize only once

// and only for expected ticks (ones that go after "Last" 2 times in a row))

return;

}

// case 3 of 3: It is normal LAST_SIZE tick telling us about new trade with the same price as before! We got new trade!

Trade * new_trade = new Trade(account, epoch_ms, tickerId, last_trade->price, size, true, 0, false);

ptrades->push_back(new_trade);

// forward complete trade to all subscribers

gotTrade(new_trade);

return;

}

…

}

[Q] Building live bars

[A]

https://groups.io/g/twsapi/topic/4046604

https://groups.io/g/twsapi/topic/4046154

[Q] Why do we get two identical tickSize() callbacks?

[A] explained by Richard L King

Date: Sat, 5 Feb 2011 00:47:45 -0000

Subject: RE: [TWS API] Re: tickSize(): LAST_SIZE and VOLUME order

Perhaps I can throw a little light on this subject to explain what is

actually happening here and why.

In the early days of the API the TICK_PRICE protocol message (ie the message

itself, not the callback function/event) had a version number of 1 and had a

price field, but no size field.

So at that time, all prices came in TICK_PRICE messages and all sizes came

in TICK_SIZE messages. Bear in mind the sampling mechanism that IB uses,

which only sends values that have changed during the previous sampling

interval. If a price had changed, it would send a TICK_PRICE message before

the corresponding TICK_SIZE message. But this approach had a problem,

because a TICK_SIZE message was only sent if the size had changed. Thus

there was no way of the client knowing, when a TICK_PRICE message arrived,

whether the corresponding size was the same as the previous size (in which

case there would be no TICK_SIZE message) or whether it too had changed and

a TICK_SIZE message would follow.

To solve this problem, IB introduced version 2 of the TICK_PRICE message in

API v760 Beta (my copy of this is dated Feb15 2004). This included a size

field, and now whenever a TICK_PRICE field was sent (which meant a change of

price during the sampling interval had changed), the size could be included

whether or not the size itself had changed. The TICK_PRICE message would now

generate both a tickPrice() callback followed by a tickSize callback. This

meant that clients had it nice and easy: when you got a tickPrice() callback

you just had to remember the price, and any following tickSize()s would

implicitly relate to that price until the next tickPrice().

However there was now a different problem. Any client that was using an

older version of the API with client version less than 15 would not

understand the version 2 message and so TWS would continue to send version 1

TICK_PRICE messages to those clients. But this meant that the tickSize()

callback following the tickPrice() callback would not be generated, so the

client would not have the full information. To fix this, after sending the

TICK_PRICE message, IB decided to also send a TICK_SIZE message with the

same size as the TICK_PRICE message. Thus clients with version less than 15

would now receive the full information.

The stupid thing was that IB should have only sent the extra TICK_SIZE where

the client version was less than 15, but they actually sent it to all

clients, and of course they still do. So now we have this silly situation

where after every TICK_PRICE message a redundant TICK_SIZE message is sent,

resulting in a tickPrice() callback followed immediately by two identical

tickSize() callbacks.

So why don't IB fix this? Well, I don't really know. It may be they think

it's of no significance, because for example anyone watching the data

onscreen would never notice. It's only because we like to tally things up

and compare with the cumulative volume figures that it irks us (and of

course because it's just plain stupid!).

And once you understand what's going on it's easy enough to code round it,

as Mike has already pointed out below.

If anyone wants to complain to IB about this and persuade them to fix it, by

all means have a go, but I very much doubt whether they'd show any interest

at all. They haven't for the past 7 years (and after all they themselves

must have had to code round it in TWS).

Actually the real annoyance here is not so much the issue itself, as the

fact that so many people spend so much time discovering it, worrying about

it, and inventing workarounds. But then that's the case for so many aspects

of the API…

Richard

[A] explained even more by Richard L King (found here)

Added on 09-Jun-2015

Let’s summarise what’s going on here:

1. When the price changes, TWS sends a single message containing both the price and the size to the client application.

2. IB’s API code then makes separate tickPrice and tickSize callbacks (note that the latter is immediately after the former – you can easily see this in the API code.

3. IB’s servers then send the size again, and the API makes another identical tickSize callback (the reasons for this are explained in my contribution to the FAQ linked to by rholowczak’s post on 2 June – this is the part highlighted in blue: I suggest you read that carefully).

4. Each time there is a trade with the *same* price but a *different* size, another tickSize callback is made.

5. When there is a trade with the *same* price and the *same* size, no tickSize callback is made.

From this you can immediately deduce that there will be *at least* 2 tickSize callbacks per tickPrice callback, and usually more because the price doesn’t change with every trade.

By the way, there is no need to fiddle around with ‘blanking time’ that you mentioned in your previous post. If you get a tickPrice callback, just record the price. The very next callback is *always* the tickSize (check the API code to convince yourself that this is true), so record the size and call your application code to process the combined price/size. Subsequently whenever you get a tickSize callback, if the size is the same as the last size you recorded, just ignore it (because it will be the duplicate tickSize described in 3 above), otherwise call your application code to process the combined price/size where the price is that recorded in the latest tickPrice callback.

It may be worth pointing out that in spite of what I said in 5. above, there *are* occasions where a tickSize occurs where the size is the same as the previous one and the price has not changed – however they are rare, and given the approximate nature of the data feed it’s not worth trying to complicate things to try and detect them. Having said, that my own API code does complicate things somewhat, but I’m not sure what I gain by doing so. So I won’t bother to describe these extra wrinkles unless someone is really interested.

[Q] Could someone suggest a good algorithm for receiving market data for more than 100 symbols with a single account?

[A] by Jason Sapp

I use a "round robin" type of algorithm that utilizes one of my paper trading

accounts to obtain cumulative volume on about 1000 symbols. I am not too

concerned if my cumulative volume number is off by a couple minutes, so this

works for me. The way I do it is as follows:

-Call reqMktData (which contains cumulative volume) for the first 90 symbols in

my list (making sure not to call reqMktData more than 30 times per second—at

least I think 30 was the magic number, but this limitation may longer be in

place).

-Wait 10 seconds (after calling reqMktData on my the last of the 90 symbols).

-Call cancelMktData on all of those symbols.

-Obtain the next 90 symbols from my list.

-Start over.

The reason I used 90 symbols was because I sometimes had a market row or two

being used by my paper trader TWS.

Jason

————————- also alternative ———————————

from here

I use Yahoo Financial's CSV feed b/c I can request multiple symbols within a

single request to get quick price data snapshots for each symbol. For

example, to request last price, daily high, daily low, close, and volume for

2000 symbols, it takes like 10 seconds to get that information from Yahoo.

I use this specifically in the morning when scanning through stocks. This

is more efficient, quicker, and reliable than trying to round-robin my way

through TWS requests and dealing with the request limits they have on their

accounts.

… by Brant Hahn

[Q] data links drop; feeds fade. Serious software needs to handle that. How?

[A] implementation idea by Russ Herrold

As a practical matter, by subscribing to some indicator that

is continuously updated as well as a more leisurely symbol,

one always has some value appear during that second when the

link is live — watch ES or such 😉

[Q] Request/response/error architecture thoughts?

[A] by Kurt (from this thread: C++ architecture question)

There are hundreds of ways of doing it. I don't believe in "ideal". I'm posting some code here because it might be useful for some others as well as you. (From this code you could probably reproduce a big chunk of the request/response/error architecture I use.)

I implemented request tracking classes with an instance per request. The default behavior is for the constructor to send the request but it is also possible to create the request and defer sending it. The class provides a bunch of useful functions including tracking latency, logging, and matching responses and errors to requests.

For this purpose I have an ActiveRequest class with subclass ActiveRequestWithContract for contract-based classes, adding additional information to the generic logging capabilities provided by ActiveRequest. Then I further have a separate subclass for each IB request type which is either a subclass of ActiveRequest or of ActiveRequestWithContract. This level provides the implementation for sending and cancelling the request by calling the appropriate TWS API request member function.

I use further subclassing for each distinct purpose or for distinct interfacing requirements, such as dispatching into Objective-C handlers.

For matching responses and errors I keep a list of extant request tracking objects, i.e. those objects that have not been deleted, which is an independent consideration from whether they have been sent, have received a response or a full set of responses, etc. The ultimate client of the request can determine whether the request object should be deleted when the response is complete, if applicable.

As a result there is a list of existing requests which can be searched by request id. This permits incoming responses and incoming errors that reference request id's to be routed to the appropriate request tracking object. Thus request clients can be notified when a request is aborted due to an error.

This is not optimal, so perhaps deficient and certainly not ideal. If performance were an issue the list of requests could instead have been a B-tree, or even a hash table. So you could use the associative arrays provided by the C++ STL.

In any case the bulk of request tracking implementation is quite removed from the tickPrice callback. However for each request tracking subclass that is specific to a particular IB request type, I have a method (member function) with name and parameters identical to the corresponding EWrapper member. I just copy/paste the function prototype. Thus a part of the class hierarchy looks like this:

MarketDataRequest : ActiveRequestWithContract : ActiveRequest

and while my EWrapper subclass of course has this member

virtual void tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute);

my MarketDataRequest class also has

virtual void tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute) = 0;

for which the implementation is left to the next-level subclass. This approach gives me a generic interface to the TWS API with none of the specifics of my app present in it.

The generic tickPrice implementation then does a linear search of a doubly-linked list to find the request object to route to, and looks like this. The superclass of MyWrapper provides a default implementation that just logs the request. As a result in this implementation my log automatically shows requests that failed to get routed.

void MyWrapper::tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute)

{

for (ActiveRequest *req = marketDataReqTracker.Next(); req != &marketDataReqTracker; req = req->Next())

{

if (req ->ReqId() == tickerId)

{

if (MarketDataRequest *specificReq = dynamic_cast<MarketDataRequest *> (req))

{

specificReq ->tickPrice (tickerId, field, price, canAutoExecute);

return;

}

inherited::tickPrice (tickerId, field, price, canAutoExecute); // default logging used when response not routed above

}

It is not perfectly efficient, but it is "perfectly adequate" for my purposes and easy enough to upgrade if needed even though I am not using container abstractions (STL or otherwise).

The use of dynamic_cast is strictly unnecessary, because marketDataReqTracker should reliably contain only MarketDataRequest objects. So this code could have benefited from some template use. But even without that a static_cast would have been safe and only because the performance hit is no issue I just did the conservative thing. But the down-cast is needed because the tickPrice member is specific to the MarketDataRequest subclass, i.e. the entire EWrapper interface is definitely not contained in each request object, per the description above.

Meanwhile error handling somewhat parallels this, with my EWrapper subclass providing this implementation

void MyWrapper::error(const int id, const int errorCode, const CString errorString)

{

(void) error_Route (id, errorCode, errorString);

}

and error_Route special-cases a bunch of error codes for more sensible and direct routing and otherwise calls ReqTracker_error_Route:

ActiveRequest *MyWrapper::error_Route(const int id, const int errorCode, const CString& errorString)

{

if (errorCode == 505)

// actual: "request 2104 error 505 Fatal Error: Unknown message id."

printf ("generic error %d %s [id=%d]\n", errorCode, errorString.CStr(), id);

else if (id == kIBNullReqId)

printf ("generic error %d %s\n", errorCode, errorString.CStr());

else

printf ("request %d error %d %s\n", id, errorCode, errorString.CStr());

bool handled = false;

ActiveRequest *handledBy = NULL;

switch (errorCode)

{

// errors specific to Orders

case 103: // [from doc] Duplicate order ID.

case 104: // [from doc] Can't modify a filled order.

case 105: // [from doc] request 45 error 105 Order being modified does not match original order

case 106: // [from doc] Can't transmit order ID:

…

case 121: // [from doc] Invalid BD flag for the order. Check "Destination" and "BD" flag.

case 122: // [from doc] No request tag has been found for order:

// 123, 124 vague

case 125: // [from doc] Buy price must be the same as the best asking price.

case 126: // [from doc] Sell price must be the same as the best bidding price.

case 129: // [from doc] VWAP orders must be submitted at least three minutes before the start time.

case 131: // [from doc] The sweep-to-fill flag and display size are only valid for US stocks routed through SMART, and will be ignored.

case 132: // [from doc] This order cannot be transmitted without a clearing account.

case 133: // [from doc] Submit new order failed.

case 134: // [from doc] Modify order failed.

case 135: // [from doc] Can't find order with ID =

case 136: // [from doc] This order cannot be cancelled.

case 137: // [from doc] VWAP orders can only be cancelled up to three minutes before the start time.

// 138, 129 vague

case 140: // [from doc] The size value should be an integer:

case 141: // [from doc] The price value should be a double:

case 142: // [from doc] Institutional customer account does not have account info

// 143 vague

case 144: // [from doc] Order size does not match total share allocation. To adjust the share allocation, right-click on the order and select “Modify > Share Allocation.”

// 145 vague

case 146: // [from doc] Invalid trigger method.

…

case 2102: // [from doc] Unable to modify this order as it is still being processed.

case 2109: // [from doc] Order Event Warning: Attribute “Outside Regular Trading Hours” is ignored based on the order type and destination. PlaceOrder is now processed.

// actual: Order Event Warning:Attempted modify of OutsideRth will not be done. Order modify now being processed.

// All the errors listed above are thought to be specific to Orders based on the message description.

// It is sometimes unclear from the message whether the order id will be available with the error, or not.

// Only errors with an id can be meaningfully routed to Order_error_Route.

// All the rest might as well be routed right away to ReqTracker_error_Route.

// In any case the above list of errorCode's are treated as a list of what should be preferentially routed to orders before request trackers.

// However the consequences of including an extraneous errorCode in the above list are non-existent for errors that turn out not to include a request id.

// For errors with a request id, an extraneous errorCode in the above list is consequential only if it is NOT an order error,

// but the request id coincidentally matches a currently-tracked order id.

// It is hard to comment meaningfully on how obscure that is.

// Instead it suggests motivation for a redesign that keeps order id's in their own numeric space,

// or interleaves them in the same numeric space with other request id's.

… rest of long comment omitted

if (id != kIBNullReqId)

handled = Order_error_Route (id, errorCode, errorString);

else//10/14/12 was: if (!handled)

// So both kIBNullReqId errors and errors that are not successfully routed to an order based on the id will end up here.

handledBy = ReqTracker_error_Route (id, errorCode, errorString);

// see comment in default case re ReqTracker_error_Route details

break;

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

case 326: // error -1 326 Unable connect as the client id is already in use. Retry with a unique client id.

clientIdInUse = true;

break;

case 505: // actual: "request 2104 error 505 Fatal Error: Unknown message id."

// Do not route this message–there should be no use in doing so since the connection is about to be closed.

// Also see special case at top of this function.

break;

case 2100: // actual: New account data requested. API client has been unsubscribed from account data.

handledBy = accountUpdateReqTracker .error_Route (id, errorCode, errorString); // make sure AccountUpdateRequest::error_Generic gets this error

AccountUpdatesCancelledByServer (errorString);

break;

default:

handledBy = ReqTracker_error_Route (id, errorCode, errorString);

// calls ActiveRequestCategoryManager::error_Route which calls ActiveRequest::error_Route for each outstanding request

// which calls error_SpecificRequest for request with matching id if found

// and may call error_Generic which is currently only implemented by AccountUpdateRequest

// for error 2100 "New account data requested. API client has been unsubscribed from account data."

if (handledBy == NULL)

if (id != kIBNullReqId)

(void) Order_error_Route (id, errorCode, errorString);

break;

}

return handledBy;

}

and otherwise calls ReqTracker_error_Route which is also less than ideal but perfectly adequate and does this.

ActiveRequest *MyWrapper::ReqTracker_error_Route (const int id, const int errorCode, const CString& errorString)

{

if (ActiveRequest *routedTo = historicalDataReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = marketDataReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = contractDetailsReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = realTimeBarReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = accountUpdateReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = marketDepthReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = fundamentalDataReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

if (ActiveRequest *routedTo = scannerReqTracker .error_Route (id, errorCode, errorString))

return routedTo;

return NULL;

}

which in turn calls this

ActiveRequest *ActiveRequestCategoryManager::error_Route(const int id, const int errorCode, const CString& errorString)

{

// Route all errors to all objects, until/unless an object's error_Route returns non-NULL, indicating it has "handled" the error.

// In a typical implementation, generic errors (id == -1) will end up being routed to all objects,

// and specific errors will be ignored except by the object with a matching request id (if any),

// which will "handle" the error and return non-NULL (usually pointer to itself), preventing further looping here

// (and in current implementation preventing equivalent of further looping in MyWrapper::ReqTracker_error_Route which calls this function for each request category until it is "handled").

for (ActiveRequest *req = Next(); req != this; req = req->Next())

{

if (ActiveRequest *routedTo = req ->error_Route (id, errorCode, errorString))

return routedTo; // very likely routedTo will always equal req

}

return NULL;

}

and there the generic implementation of error_Route comes into the picture

ActiveRequest *ActiveRequest::error_Route(const int id, const int errorCode, const CString& errorString)

{

if (this->ReqId() == id)

{

error_SpecificRequest (id, errorCode, errorString);

return this;

}

if (id == kIBNullReqId)

{

if (errorCode == nonAPIErrorCode_connectionClosed)

Aborted (errorCode, errorString);

else

error_Generic (errorCode, errorString); // as of 5/3/12 this is the only call to error_Generic

}

return NULL;

}

And an implementation of error_SpecificRequest is provided by the request-tracking subclass for each API request type, such as this one.

void MarketDataRequest::error_SpecificRequest (const int id, const int errorCode, const CString& errorString)

{

if (errorCode == 200)

{

Aborted (errorCode, errorString);

}

/* market data error reference

error 39 200 No security definition has been found for the request

the same error can happen for reqMktData and reqContractDetails and reqHistoricalData

in this case it was a bogus combo (spread between AAPL and INTC options)

sending MarketDataRequest 39 AAPL,INTC SPREAD

but the "No security definition" error should be dispatched to the final request object

and should maybe delete without cancel

are there existing cases in which [??? text lost ???] caused an unintended cancel?

}

So that may give you some sense for an approach. While not ideal there is a lot of architecture there that supports quite a bit of experience working with the API.

-Kurt

[Q] Thoughts on “duplicate requestID” value (bad value given by API)

more details:

In my program, I get the NextValidId on connection to TWS and I use this

number to place my first order for the day, and I keep incrementing to place

the orders following it.

It so happened yesterday that the NextValidId returned by ID was a duplicate

one. The program had placed an order with order ID 638 few days back and It

placed the another order with the ID, 638.

(both the orders got executed)

I have unchecked the option "Reuse rejected order" in TWS — orders

configuration menu.

Does anyone have a clue why was an used OrderID returned by IB?

Thanks,

Vishruth

[A] by Frank Bell

I found it necessary to keep my own persistent copy of the current order

number. At start up I use max(my number, NextValidId).

Frank

[A] by Richard King

TWS stores the next valid id in its settings file. This is a large XML file,

and I suspect it's only written out when you shut down TWS (though I think

there is a save settings command in the menu? – can't check this at the

moment).

So if TWS crashed or you killed it rather than tidily closing it down, that

might account for your observation.

Richard

[A] by Jan Boonen

I also check the orderId of the EWrapper::openOrder( OrderId orderId, …) events, just in case of. If that order id is equal or bigger then the one to be used next, I set the next available value to orderId+1.

Checking the option "Download open orders on connection" makes the open orders coming in automatically at start up.

janB

[Q] Thoughts on “requestID space partitioned”

Hi Kurt,

Thank you for shading more light on the architectural constructs you developed.

Most definitely I'll reuse your approach of having one tracking instance for each individual request along with searchable lists (or maps as you suggested) to quickly retrieve corresponding objects for proper response routing/handling.

I was developing something very similar couple weeks before and part of the logic was automatic request ID generation at which point I recall that few months ago I read your take on request ID from some of the tread I can no longer easily find. If I recall correctly you said you track order request IDs separately from all other kinds of requests.

If it is the case, then my question is: how would you distinguish between "order's request IDs" space and "other request IDs" space if they overlapping? Example: you got error() call with ID matching one order tracking set and one from "other requests" set. Then route became ambiguous.

When I further thought about this problem I thought it would be generally good idea to add "expiration" field to each request, which can be set to something non-zero for requests with finite lifetime expectancy or to zero value for non-expirable ones (like for data subscription – should live "forever" until canceled). Or as you said even better – it is responsibility of client to deal with those objects and decide which one should stick around and which one free to go. But even hypothetical short life cycle of request tracking instances does not 100% complete less often "ambiguous route" cases if we try to separate orders from others. (and why do we want to do this in the 1st place).

ps: other possibility (if we need to separate orders from others by ID) would be to use odd numbers for orders and even for other requests, circling those spaces (like process ID assigned in linux OS:), but its ugly and unnecessary shrinks space for IDs twice (and I doubt we need same power of sets for orders vs other requests)

Thank you,

Dmitry

[A] by Kurt (from this thread: C++ architecture question)

[snip]

Mind you just because I have so far gotten away with this approach doesn't mean it is the best or that I will continue to get away with it. I like to action on issues that are not pressing so I can keep considering different angles, or … maybe leave it as indefinitely. Others have partitioned the requestID space. I think Richard has done this. It has been pointed out that it is a plentifully big space.

I just happen to like small numbers so I use small numbers for order IDs and small numbers for other request IDs. Your odd/even idea (below) would also take care of this although I have the memory it may cause problems for certain kinds of orders for which IB may automatically insert another orderID on your behalf at the next integer value.

The reason I get away with it is that the error code space is somewhat defined by IB. The MyWrapper::error_Route function which I mentioned but did not show has, among other things, a big switch on errorCode and a long list of errors known to be particular to orders, based on the IB documentation. Here is that function with the long list of order-related errors truncated.