Dmitry’s TWS API FAQ

Share this post

Dmitry’s TWS API FAQ

Last updated on: 27-Jun-2020

Table of contents:

About this FAQ

Purpose

Scope

Disclaimer

Link to the latest IB API documentation

TWS app related

[Q] How many market data lines one has subscribed to in TWS?

[Q] How to import Tickers from a File?

Import Tickers from a File

[Q] Link to the official server reset time?

[Q] How to check and solve connectivity issues affecting the Trader Workstation (TWS)

[Q] How to run 2 TWS on different machines (to access same account)?

[Q] How to optimize TWS Java VM performance? aka the "latest and greatest" JVM server settings

[Q] How to make TWS application running 24/7 automagically?

Optimal memory setting

[Q] How to setup IBController (3.2.0) + TWS (build 956)  on headless Ubuntu 16.04 LTS to run TWO accounts (paper + real) in 10 minutes?

[Q] Where do we get the latest versions of TWS API?

[Q] How to reset TWS connections?

[Q] which version of TWS should I use?

[Q] How to reset paper account?

[Q] How API version maps to client/server version I see in a log?

[Q] Client can not re-connect to TWS using same client ID after connection was not properly closed. Wtf? And how to solve?

Code Examples

[Q] Where can I find C++ and C# tutorials/examples using ActiveX?

[Q] Where can I find working posix C++ code examples?

[Q] Where can I find MORE ADVANCED working posix C++ examples?

[Q] Python?

[Q] Delphi?

[Q] Java?

[Q] ruby?

[Q] C#?

[Q] R

[Q] Pattern recognition library?

[Q] Reverse engineering java?

Implementation notes

[Q] What value means "no value"?

[Q] Best practice to handle ID’s / nextValidId?

[Q] After connection lost I need a list of all open orders and positions in order to get synchronized?

[Q] Tracking time (internally)

[Q] Building live Trades

[Q] Building live bars

[Q] Why do we get two identical tickSize() callbacks?

see also somewhat related:  [Q] Building live Trades

[Q] Could someone suggest a good algorithm for receiving market data for more than 100 symbols with a single account?

[Q] data links drop; feeds fade. Serious software needs to handle that. How?

[Q] Request/response/error architecture thoughts?

[Q] Thoughts on “duplicate requestID” value (bad value given by API)

[Q] Thoughts on “requestID space partitioned”

[Q] Trying to modify an order and sometimes FAIL!?

[Q] Trying to modify trailing stop, got an error 10067

[Q] Order execution messages considerations

[Q] Checking API connection / checking the timeliness of ticks

[Q] AvgPrice rounded to 2 digits, I want 3 digits!-)

[Q] How to get avgFillPrice for multiple filled order status updates?

[Q] rounding to the tick size of the Financial Instrument

[Q] Should I use MKT or LIMIT orders? How do I track partial fills?

[Q] Why PosixMQ is a good idea for IPC? Why can’t I just use some thread-safe queue (with mutex)?

[some feedback from 100% complete frameworks out there;-]

[Q] Unit testing – which one to choose?

[other chaotic implementation notes]

Well known errors and how to avoid them

[Q] reqSecDefOptParms() with exchange specified returns no data?

[Q] I observe gaps in the most recent historical date (options in my case). But later same data looks good. Why?

[Q] Errorcode(110): The price does not conform to the minimum price variation for this contract

[Q] Errorcode(162): Error message:No market data permissions (for some tickers, even with proper subscription!)

[Q] Errorcode(162) Historical Market Data Service error message:HMDS query returned no data: GCQ9@NYMEX Trades

[Q] Errorcode(162) 'Historical Market Data Service error message:Starting time must occur before ending time'?

[Q] Errorcode(200) No security definition has been found for the request

[Q] Errorcode(201): Order rejected – reason:It does not comply with IB's order handling rules for derivatives that result in physical delivery.

[Q] Errorcode(300): Can't find EId with tickerId:40000001.

[Q] Errorcode(321): Missing exchange for security type FUT

[Q] Errorcode(321): Error validating request:-'rc' : cause – You must specify an account.

[Q] ErrorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – Historical data queries on this contract requesting any data earlier than 199 year(s) back from now which is 18210405 00:18:14 GMT are rejected.  Your query would have run from 5240807 23:29:46 UTC to 5240807 23:33:06 UTC.'

[Q] errorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – End date not supported with live updates'

[Q] Errorcode(???) Order Canceled – reason:Order size exceeds amount allowed by fat-finger check

[Q] Errorcode(10225): Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately

[Q] Errorcode(1101, 1102): Error validating request:-'rc' : cause – You must specify an account.

System Message Codes

[Q] Error: 2109 Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed.

[Q] Order size is too large IB limit is 35000

[Q] First order after IB GW API connection causes disconnect and order is ignored

[Q] Historical Market Data Service error message:No historical market data for XAUUSD/CMDTY@CMDTYBBO Last 60

[Q] While trying to modify submitted order getting: "Unable to modify this order as it is still being processed."

ContractDetails

[Q] Exchange list in ContractDetails?

Placing Orders (common)

[Q] I sent an order, what should I expect?

[Q] How do I query the status of my orders?

[Q] How to distinguish Client ids from API vs non-API?

[Q] Where to Set GTC in placeOrder API?

[Q] How often can I modify an order?

[Q] This account cannot have open orders(including Combo) on both sides of the same US Option contract in related accounts.

[Q] How to change the limit price of an order without changing order size after partial fills?

[Q] I’m getting multiple filled orderstatus updates

[Q] Ok! Order executed! But what was the fill price?

[Q] Order Id mismatch between API and TWS

[Q] Why my order has “Inactive Order” Status?

[Q] How to cancel all orders?

[Q] How to cancel all open orders and close all positions?

[Q] How to pause order (like in TWS)?

[Q] How to get order history?

[Q] How to track ALL orders (including cancelled orders, and including manually created via TWS when my program was not running)?

[Q] I got multiple execution messages.

[Q] How to submit a sell order as soon as a buy order is executed

[Q] How do I know if my stop order is triggered from API message?

[Q] Should I use IB’s trailing stop and other advanced order types?

[Q] OCA orders?

[Q] OCA orders are simulated orders?

[Q] Code: 201 – Order rejected – reason:The OCA group order has already been filled

[Q] Order overfilled?

[Q] SHORT selling (SSHORT vs SELL)?

[Q] Changing Order Quantities while order partly filled.

[Q] How do I get commissions from API?

[Q] Also if I want to see the sum of Commissions in a given day from API ?

[Q] How to check the date when the current position was acquired?

[Q] Can I use just conId for specifying the contract?

[Q] My order’s place in line!?

[Q] Max usage of buying power for DT system

[Q] openOrder / orderStatus / execDetails sequence?

[Q] Orders pacing?

[Q] How to modify submitted order?

[Q] error: The price does not conform to the minimum price variation for this contract

[Q] Sometimes reqOpenOrders will not return all open orders. wtf?!

[Q] I have not able to retrieve the previous filled orders through the reqExecutions call

[Q] How to submit a vwap order with start and stop times and other options?

[Q] Relative Order transmit problem

Bracket orders

[Q] Bracket order: Why Sell and Stop don’t cancel each other?

[Q] Bracket orders: 3 or more childs? grand child? stop order as an entry?

[Q] Bracket orders: edge cases

[Q] Bracket orders: How to place bracket order via API (recap)

Placing Orders (stocks)

[Q] How can I keep a reference to these manual trades when they pop up in openOrders if all the IDs are 0 at the time?

[Q] What "Order Inactive" really means?

[Q] How to send a limit order then if the order is unfilled after a certain amount of time have the order be cancelled and do market order?

[Q] order.m_account = null ?!

[Q] m_shortSaleSlot's field?

Placing Orders (options)

[Q] How to get daily volume for options?

[Q] How to Monitor Stock Loan Availability?

Monitoring Stock Loan Availability

Related Articles

[Q] How to find SLB info?

[Q] TWS and Option Spreads

[Q] weird theta

[Q] How should an API client detect an option exercise, the exercise being done manually in TWS?

[Q] How to find out available strike prices for options ?

[Q] How to pull entire option chain?

[Q] Why option chain data is delayed (sometimes up to 1 minute)?

[Q] Is IB’s model reliable or useless and I need to do all the calculations myself?

[Q] Am I required to pay the dividends on shares borrowed when short selling?

[Q] Which stocks are shortable which are not?

[Q] Unable to borrow?

[Q] How to determine the pricing step for an option?

[Q] placeOrder() No security definition has been found for the request

[Q] I see duplicate events from time to time – wtf?!

[Q] attached order

[Q] Bad equity options spread (bid-ask seems to widen significantly)

Placing orders (futures)

[Q] Can I use SMART for futures?

[Q] Can I place MOC order for es future?

[Q] futures options data via API

[Q] How to get Futures Front Month

[Q] continuous contracts rollover schedule

[Q] Continuous contract – how to determine offset?

[Q] How to create a true continuous contract historic data (futures)? Do you also perform back propagation of price adjustments when you roll?

[Q] Futures: what is “First Position Date”?

Combo Orders

[Q] Placing Combo orders?

[Q] Combo order example:

[Q] Get info on all legs of placed combo order?

(quote)

[Q] Note: good-after-time (GAT) orders are not supported for ("generic") combo orders!

[Q] Can place orders for each leg, but can’t place combo!? wtf?

[Q] Is there a way to get contract details of the BAG contract itself?

[Q] Combo example please!

[Q] IB Lag with Combination Orders

[Q] Can’t change qty on combo order!

After trade is complete

[Q] How to get all Completed Trades within the last N days?

[Q] tell me more about Flex Query

Paper vs real accounts

[Q] Accuracy of paper-trading account

[Q] PaperTrading – how do they simulate fills on limit orders?

[Q] Paper account vs. Demo account?

[Q] Partial fillings of MKT order on paper trading account? wtf?

Limitations of the PaperTrader

[Q] I am wondering if there is any way to tell from the API whether the socket is connected to a paper trading account or a real account?

[Q] Trying to place pegged to stock orders through API or TWS via paper account?

Charting

[Q] Any easy way to do charting?

Fundamental data

[Q] reqFundamentalData() – How to Parse XML File?

[Q] How often Fundamental data updated?

[Q] Why only 4 out of 6 fundamental requests succeed and for 2 of 6 I got errors:

[Q] You mentioned "we no longer directly provide support for API Ver. 9.67". How does this work? Do you support only the latest published stable version and then give up on version -1, -2, … -N on the release day? What are the rules here?

IB Data Subscription

[Q] NYSE auction order imbalance data?

[Q] How to download broad market statistics, such as NYSE  TRIN, NYSE Advancing issues, NASDAQ ?

[Q] tickPrice vs tickString?

[Q] Can I get historical price data of instruments which have expired?

[Q] Trust daily close or 1-minute data close?

[Q] VWAP through the API

[Q] What RTVolume fields are?

[Q] Realtime Volume timestamp sometimes goes backwards?

[Q] Is there an easy way to get the S&P 500 index component stocks?

[Q] Can I download the historical data of obsolete symbols?

[Q] My accumulated volume does not match the TWS one! Wtf with VOLUME?

[Q] Does it make sense to use separate data / broker connections for separate tasks?

[Q] API and Trading hours?

[Q] How to get the day open price for NYSE stocks?

[Q] How to get Futures Open Interest?

Data subscription: Volume

[Q] How to get somewhat accurate cumulative volume?

Data subscription: Dividents

[Q] Get Dividend information?

[Q] Dividend Calendar (dec, rec, div, exdiv, amount, etc)

[Q] Dividend Adjustments? Adjusted Previous Close not available on IB?

[Q] API reqFundamentals dose not return previous dividend data for index

[Q] Data subscription: Calendar events

[Q] I got invalid update of bid/ask which leads to CROSSED PRICE. wtf?

[Q] Crossed quotes on IB (again)

[Q] Snapshot market data vs “real time” data.

[Q] Market Data Stuck when requesting snapshots!

[Q] reqRealTimeBars vs reqMarketData – how to get real time bars?

[Q] How to get list of all IB’s tradable contracts?

[Q] Sometimes lastSize=0. wtf?

[Q] When is exact start of a new bar? Will hand-made bars (from tick data) be exact same bars as I receive from IB?

[Q] Why IB uses sampled data instead of plain real-time tick streams? Aka "fudge" factor for tickSize event.

[Q] What are the differences between Level I and Level II data ?

[Q] Which request ID should I use?

[Q] How deep can I go in history?

[Q] Bug in historicals? Price but no volume and other historical “holes”!

[Q] Options historical data?

[Q] How to store historical data?

[Q] reqHistoricalData: Volume vs Count?

[Q] How to backtest?

[Q] high, low, close fields in tickPrice ?

[Q] What is the sequence of tickSize() tickPrice events?

[Q] I get negative and zero quotes! wtf?

[Q] How to get all strike prices for a given ticker & expiry?

[Q] No security definition has been found.

[Q] reqMarketData(): For most stocks I get the last prices, however for some stocks I only get bid.

[Q] reqMarketData(): sometimes getting back tickPrice events with a price of -1

[Q] What is conId?

[Q] Volume looks incorrect?

[Q] How to get indices historical prices?

[Q] How to get realtime P&L?

[Q] reqRealTimeBars() – "whatToShow" string?

[Q] reqRealTimeBars() – only 5sec bars available? Partial bars?

[Q] tickPrice(): canAutoExecute field – wtf?

[Q] When requesting historical data I often got “Invalid step”

[Q] Valid Duration and Bar Size Settings for Historical Data Requests

[Q] Is it possible to develop TWS like depth book using reqMktDepth() call?

[Q] What happens when I do reqMktData() afterhours?

[Q] IB's own historical data corrections (history of the history:)

TWS Scanners

[Q] Where do I begin with scanners?

[Q] Biggest gainers since open between $5 and $10?

[Q] Give me simple example of working scanner request.

[Q] Note on contracts which may not be accessible through TWS when running scanners:

[Q] Market Scanner Pre/Post Market Hours:

Basics for beginners

[Q] Vocabulary

[Q] Limitations on quantity of symbols?

[Q] Where can I find the API version info?

[Q] I’m about to post a question to IB API list. What should I do?

[Q] Number of hops to IB?

[Q] Is IB real time data actually “real time data”?

[Q] When I pass a BUY LIMIT order on the ES, let's say at 1300.25, I always have to wait until the price goes to 1300 to get a fill…is that normal?

[Q] Will I have this problem if I try my algorithm with a stock like IBM on SMART?

[Q] I am behind a firewall I like to know the TWS out bound ports?

[Q] Other TWS API FAQs?

[Q] Any general recommendations?

[Q] IB commission rate comparison

[Q] How to sync market data?

[Q] Notice of High Order Ratio

[Q] Some IB API timing requirements

Historical Data Limitations

[Q] What is Pacing Violations?

[Q] Can I be long and short on the same underlying same time?

[Q] Any historical-data aggregating programs?

[Q] How to diagnose an issue?

[Q] Why should NOT I use DDE?

[Q] Is demo account API behavior differ from the behavior of real account?

[Q] What are "EX" methods?

[Q] custom indicators?

[Q] Any other tips/hints for beginners please?

IB Gateway app related

[Q] Installing IB Gateway and TWS API for linux?

[Q] Trader Workstation (TWS) vs IB Gateway?

[Q] Any performance difference between IB Gateway vs TWS app APIs?

[Q] Gateway Crash on Startup – Linux

[Q] list of the error 321 "message is not supported"

[Q] autologin script for IB Gateway?

[Q] How to automatically accept client connections to TWS and/or Gateway from any remote IP address (not known in advance)?

Some Forex notes

[Q] IDEALPRO LMT orders at the top of the order book never executed

[Q] Please check my Forex math!

[Q] according to the documentation m_avgPrice should include commissions, but it doesn’t!!

[Q] Can't get some Forex historical data?

[Q] Why I don’t get TRADES from “real-time” FX ticker?

[Q] Tracking Forex positions?

[Q] IDEALPRO's hours?

[Q] minimum size requirements for IDEALPRO

Other

[Q] Best take profit order type

[Q] contract->exchange = "SMART" works, but contract->exchange = "NASDAQ" does not!

[Q] IB technical support contact info?

[Q] fully clarifying exchange vs primaryExchange

[Q] I need some actively trading stock for my midnight debugging

[Q] Where do I get the latest standalone TWS?

[Q] How do I know if the market is open via API?

[Q] How to get account balance, transaction fees, cash transactions, Net Asset Value, etc.? (Flex?)

[Q] How to automate Flex Reports?

[Q] Which Java is better to use for TWS on Ubuntu?

[Q] Is there any way to see S&P 500 index on TWS?

[Q] Release notes – should I read it?

[Q] French transaction tax, how to anticipate?

[Q] Getting exchange location for order?

[Q] What is the "numIds" for in reqIds function?

[Q] Does anybody know of a free technical analysis libraries?

[Q] Is there a way to pull these daily libor rates directly from TWS?

[Q] Closing Prices in TWS Are Wrong?

[Q] How to place MOO orders?

[Q] How to update the future's margin requirement from TWSAPI?

[Q] How to get margin requirements for possible trade?

[Q] [TWS API] Stress-testing margin requirements using Risk Navigator to model changes in the underlying price?

[Q] Figuring out Lookahead / Overnight Margin Rate For A Stock

[Q] index of all available addons that can be purchased for tws?

[Q] reverse engineer IB API socket protocol

[Q] Read this later for TAX purposes (if you ever need it at all)

[Q] Can I get the MarketValue for all the currencies?

[Q] Managed Accounts?

Reporting

[Q] How Do I Match the OrdeId that is sent with the Executed Order ID that appears in the Trades and Summary tabs?

[Q] orderStatus function is be called 3 or 4 times(!) even when there is no update of status. Wtf?!

[Q] How to track order fills?

[Q] How to obtain the complete current portfolio?

[Q] Is there a way I can get total short value of my account via the API? I'm basically looking for the total value of all short positions.

Not yet confirmed (but possible issues)

[Q] Validity of Paper Account for Testing Algo Trading Outcomes?

[Q] Other limits I’d be aware of?

[Q] there is no correlation between bid/ask and trades in IB's feed (?!!)

[Q] updatePortfolioEx() non consistent behaviour?

[Q] Access multiple accounts?

[Q] How to monitor events (keyboard, mouse) in X?

APPENDIX

Time drift

[Q] Does my/your/any computer have a time drift?

[Q] Should I build own NTP server (on my LAN) or would it be sufficient to just use public ones?

[Q] How to build own Stratum-1 NTP server on Raspberry Pi?

Some unsorted yet related stuff


About this FAQ

Purpose

Let us learn and make TWS API better and easier to use/understand.

This document is Dmitry’s personal attempt to learn TWS API and 99.9% of its content came from TWS API Group https://groups.io/g/twsapi. I tried to structurize some questions and answers in form of FAQ grouped by topics.

Scope

 I tried to collect all the general Q/A’s about data flow, tricks of certain functions of TWS API, limitations, workarounds and other general-level info. On the lower-level side of things (language specific) I collected mostly C++ / Java related topics and did not store much on topic of Windows Sockets, ActiveX, CSharp, VB or DDE for Excel etc. Want to add missing topics? You’re welcome to participate and add it yourself!  πŸ™‚

In Dec-2014 I started digging (reading through) all the history starting from messages around 05-Feb-2010 and this FAQ is ongoing attempt to keep track on all important topics.

Disclaimer

This FAQ represent a small fraction of the TWS API Group discussions and by no means is intended to replace one. Please search online group archives for much more answers (than you can find here).


Link to the latest IB API documentation

[A] by Josh

On Wed, May 3, 2017:

The newer documentation is at: http://interactivebrokers.github.io/tws-api/ 

by the way.

Josh

–also–

———- Forwarded message ———-

Date: Thu, 18 May 2017 17:29:38

From: jbeacomib <notifications@github.com>

Reply-To: InteractiveBrokers/tws-api

To: InteractiveBrokers/tws-api <tws-api@noreply.github.com>

Subject: gh tws-api]  Stale IB doco relative to the github  content (#493)

The documentation which is actively updated with the most

current information about the builds of TWS and API versions

that are on the website (API versions 9.72, 9.73) is hosted on

[Github](http://interactivebrokers.github.io/tws-api/)

The older documentation was created for version 9.71 of the

API and previous builds of TWS. It is now no longer updated

and has been superseded by the Github documentation site. The

new documentation is correct that up to 32 API applications

can now be connected to one instance of TWS or IB Gateway.

However in practice, this limit is uncommonly reached as it is

usually more efficient to reduce overhead with a smaller

number of connected clients.


TWS app related

[Q] How many market data lines one has subscribed to in TWS?

[A] View/Reply Online (#43856)

Added on 23-May-2020

Press Ctrl-Alt-= simultaneously on your keyboard, TWS will show pop-up window with number of lines used.

See also:

https://ibkr.info/article/3292

[Q] How to import Tickers from a File?

[A] discussed here (View/Reply Online (#43800))

https://www.interactivebrokers.com/en/software/tws/usersguidebook/getstarted/import_tickers_from_a_file.htm  Here's copy-pasted article from IB's website in case the link above no longer works:

Import Tickers from a File

You can populate a trading page with market data that you import from a comma-delimited .csv file or .txt file. Create the list using any word processing program and save the file with a .txt file extension, or in MS Excel and save as a .csv file.

To create file of ticker symbols to import

  1. Create a file with the extension .txt, for example twsTickers.txt, or create a .csv file in Excel.
  2. In the text file, create one comma-delimited line for each line of market data you want using the syntax described below. In Excel, create one row, using one cell for each field. Note that the first field designation is the line type: DES, SYM, and CON.

DES, UNDERLYING, SECTYPE, EXCHANGE, LASTTRADINGDAY, STRIKE, PUT/CALL, MULTIPLIER

SYM, SYMBOL, EXCHANGE

CON, CONID, EXCHANGE 

The DES line type requires information or placeholders for information, with the strike, Put/Call and multiplier fields being optional. Format for the Last Trading Day is YYYYMM, or YYYYMMDD. All entries must be in caps. An example for an options contract XYZ would be:

DES, XYZ, OPT, ISE, 201509, 75, CALL,

A stock instrument for symbol XYZ in this line type would look like this:

DES, XYZ, STK, SMART,,,,

where the commas are placeholders for fields you don't define for stocks.

Using the SYM line type for a stock would look like this:

SYM, XYZ, SMART

Use the CON type to quickly enter contracts with a conid, for example

CON, 12348765, SMART

To import a file into Classic TWS

  1. From the File menu, select Import/Export and then select Import Contracts.
  2.  
    Use the Browse button to find the file to import.
  3. Specify where you want imported tickers to go, either onto an existing page or onto a new page.
  4. Click OK.

To import a file into a Mosaic Watchlist

1.  Right click in the Watchlist, and select Import/Export then Import Contracts.

2.  Use the Browse button to find the file to import.

3.  Click OK.

Imported contracts entered onto an existing page appear at the bottom of the active trading page or Watchlist.

[A] by Josh

typing a local symbol directly into an watchlist row will bring it up immediately, because it is unique to that option

i.e. 'AAPL  200221C00322500' (make sure you have the correct spacing)

Option Symbology Initiative (OSI) standard: https://ibkr.info/article/972

[Q] Link to the official server reset time?

[A] by Josh

https://www.interactivebrokers.com/en/index.php?f=2225

[Q] How to check and solve connectivity issues affecting the Trader Workstation (TWS)

[A] by Josh

https://ibkb.interactivebrokers.com/article/2807

(I’ll preserve a quote from the link above just in case here;)

Background:

The Trader Workstation (TWS) software needs to connect to our gateways and market data servers in order to work properly. Connectivity issues affecting your local network or your Internet Service Provider network may negatively affect the TWS functionality. In this article we will indicate how to test your connectivity using an automated connectivity test web page.

 

How to test the connectivity using the automated "IB Connectivity Test" web page?

1) Click on this link: http://www.interactivebrokers.com/cgi-bin/conn_test.pl 

2) Please wait until all the tests have been completed and results have been displayed. If you see "Success" as outcome for all tests, your connectivity to IB Servers is reliable at the present moment. No additional connectivity troubleshooting or configuration should be needed.

3) If you see "Fail" as outcome for one or more test/s, please click on the link "Fail" itself in order to display the "IB Network Troubleshooting Guide". That section will help you conduct some manual tests to identify the cause of the failure.

Note for Corporative environments and Proxy server users: the automated "Connectivity Test" page may return misleading results in case your machine is accessing the Internet through a Proxy server. This usually happens if you are connected to a company network. If this is your case, we kindly ask you to contact your Network Administrator or your IT Team and ask them to perform a manual connectivity tests towards the destination servers indicated in the table on the top of the IB automated "Connectivity Test" web page itself. The manual connectivity test should be conducted using destination TCP ports 4000 and 4001. Should they prefer to have the server list in another format or should they need to set up the firewall / IP Management rules, you can forward them this page.

(end of quote)

[Q] How to run 2 TWS on different machines (to access same account)?

This question was posted on github ib-controller project:

Hi!

Any chance you can add a setting to deal with the "someone has logged in with this username from another location" scenario?

IB's "great" monster doesn't let you log in from two locations so I'd like to be able to login from my phone to check my portfolio at times that I know the TWS API will not be sending orders.

I would imagine it would click the button after some timeout.

Let me know what you think!

Thanks

Georgios

Added on 16-Aug-2016

[A] by Ben Alex

One option is a second username on your account. Full info at http://ibkb.interactivebrokers.com/node/1004.

[Q] How to revert installed version of API properly? aka downgrading API

Added on 29-May-2015

[A] by Richard L King (found here)

Well, I’ve discovered why I was unable to get any previous ActiveX API version working after uninstalling 9.72 Beta..

When you install the TWS API, the install process puts a library called TwsSocketClient.dll into the Windows system folder. This library has a file version number the same as the API version, and the ActiveX control (Tws.ocx) is actually a wrapper around this library, so Tws.ocx and TwsSocketClient.dll need to be the same version for the ActiveX API to be guaranteed to work properly.

The problem arises from the fact that when you uninstall the API, the TwsSocketClient.dll is not removed. If you now try to install an earlier version of the API, the existing TwsSocketClient.dll is not replaced, presumably because the version number being installed is lower than the one already there. So now the version installed does not match what the ActiveX control expects, and so it doesn’t work – at all.

So the solution is simply to manually delete TwsSocketClient.dll before attempting to install an earlier API version.

rholowczak, perhaps it would be useful including this info in your ActiveX tutorial? Most people probably never revert to a previous API version, but they may want to if, for example, they’ve been trying out a beta version and want to revert to the latest supported version.

[A2] by newguy@videotron.ca

Added on 30-May-2015

That's actually documented by IB.

Uninstalling and Re-installing the TWS API Software on Windows

Uninstalling and Re-installing the TWS API Software on Windows

Open topic with navigation You are here: Overview > Uninstalling and Re-installing on Windows Uninstalling and Re-installing the TWS API Software on Windows

View on www.interactivebrokers.com

 

[Q] How to optimize TWS Java VM performance? aka the "latest and greatest" JVM server settings

Added on 27-Apr-2015

[A] by FutureScalper (found here)

you should use IBController by Richard L King

I would emphasize that the most significant thing you can do to improve

performance is to use the Java Server VM. I have a high performance

multi-threaded Swing-based GUI applications which does realtime analysis and

order processing, etc. It runs using the Server VM, even though it is a

"client" application. I also run TWS using the Server VM. This provides a

range of tuning options and global optimization capabilities which probably

doubles or triples throughput.

You can get the Server VM only by installing the JDK, and then you may want

to move the jvm.dll into your JRE area or whatever allows you to use the

command "java -server -version" to verify its installation.

Again, running TWS itself under the Server VM as well is critically

important for high performance.

Here is an example of the command line which I use to run standalone TWS

under the Java Server VM on Windows. Various options are technical, and

subject to change.

c:\WINDOWS\system32\java.exe -server -dsa -Xbatch -Xss192k -Xms200m -Xmx200m

-Xnoclassgc -XX:+ForceTimeHighResolution -XX:-TieredCompilation

-XX:+RelaxAccessControlCheck -XX:MaxInlineSize=64000

-XX:-DontCompileHugeMethods -XX:CompileThreshold=250 -XX:CICompilerCount=1

-XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=150 -XX:+DisableExplicitGC

-XX:ThreadStackSize=192 -XX:+AggressiveOpts -cp

jts.jar;pluginsupport.jar;jcommon-1.0.0.jar;jfreechart-1.0.0.jar;jhall.jar;other.jar;riskfeed.jar;rss.jar

jclient/LoginFrame \jts

The Server VM will compile "huge" methods, and will "inline" code, and

perform global optimizationsn not possible with the standard Client VM.

Performance and scalability matter for many applications.

[Q] How to make TWS application running 24/7 automagically?

[A] you should use IBController by Richard L King

URL: https://github.com/ib-controller/ib-controller

You can download the latest version from
https://github.com/ib-controller/ib-controller/releases/latest

(quote-1)

One possibility is to use IBController, which is a Java program that loads

TWS and intercepts various window events and handles them automatically.

Note that IBController can only handle logon if you have opted out of the

security device scheme.

You can find IBController in the Files section of this forum, under Auto

Login Codes.

Richard

(end of quote-1)

(quote-2)

IBController is a Java application that was written to enable Interactive
Brokers Trader Workstation (TWS) to be run in "hands-free" mode.  This
makes writing unattended automated trading systems possible.  IBController
automates the TWS login by filling the login dialog with your login
credentials. It also handles the dialog boxes that TWS presents
during programmatic trading activies.

(end of quote-2)

link: http://sourceforge.net/projects/ibcontroller/files/

few random notes from IBController users:

—————————————————————————————————————————————-

by Mike Smith mikesmithv@gmail.com

You might try changing the MaxPerSize parameter to something larger, like from 128M to 255M.  When that number is too small it can cause any number of weird symptoms.  When there is not enough space to load a new class then a previously loaded class is unloaded and strange things can happen.  Programmers sometimes assume static class variable are "global variables" that remain as long as the application is launched and that's not the case if the class is unloaded.

Not long ago TWS used the default MaxPerSize and that was not enough if launching under IBController.  Adding -XX:MaxPermSize=128M the the IBController launch command was needed because of the extra classes loaded by IBController.  Now I see on IB's web site the launch command for TWS now contains XX:MaxPermSize=128M just for a normal TWS launch.  So it appears they have added more classes to TWS so they had to bump up that number for a standard launch.  So it makes sense a larger number is needed for launching with IBController now.

I don't use IBController anymore so all this is based on my past experiences with it.  Now I use the gateway restarting it once a week and just let it run all week on one log in.  So all this "in theory" and not my direct experience.  I hope it makes sense.

see also discussion:

Optimal memory setting

https://groups.io/g/twsapi/topic/4047139

—————————————————————————————————————————————-

[Q] How to setup IBController (3.2.0) + TWS (build 956)  on headless Ubuntu 16.04 LTS to run TWO accounts (paper + real) in 10 minutes?

[A] https://dimon.ca/how-to-setup-ibcontroller-and-tws-on-headless-ubuntu-to-run-two-accounts/

Added on 13-Jun-2016

Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.

[Q] Where do we get the latest versions of TWS API?

[A] http://interactivebrokers.github.io/

[Q] How to reset TWS connections?

[A]

Ctrl-Alt-R resets the account server connection

Ctrl-Alt-F resets the market data connections

[Q] which version of TWS should I use?

[A] by Ruediger Meier sweet_f_a@gmx.de

Actually I install all TWS versions in parallel just to be able to try

it out whenever I will be in the right mood.

The default one which I also use for production is the stable version

936.9l.

For those who don't know it, IB provides stable and latest versions. The

current stable TWS gets only bugfix updates but no new features. That's

why it has such a high minor version number (letter).

[Q] How to reset paper account?

[A] by Eric J. Holtman

Under "Trading Access", pick "Paper Trading Account Reset".

— or —

Log into paper account management (don't log into regular account)

https://www.interactivebrokers.com/sso/Login

Left side of screen – "Trading Access" – under that – "Paper Trading

Account Reset"

It doesn't kick in until next trading day.

by Jim Covington

[Q] How API version maps to client/server version I see in a log?

[A] by IB support (by phone)

Client side:

API 9.67 —> Client version 59              TwsApiC++.9.67_3.zip

API 9.70 Release Date: October 2013 —> Client version 62

API 9.71 Release Date: Aug 6 2014 —> Client version 63 

Server side:

build 946 —maps-to—> TWS socket server version 71

[Q] Client can not re-connect to TWS using same client ID after connection was not properly closed. Wtf? And how to solve?

[A] by Dmitry

https://groups.io/g/twsapi/topic/4047690

And followed discussion in the hijacked thread πŸ˜‰

Same topic was also discussed here:

https://groups.io/g/twsapi/message/37644

Basically it is about: when client closes connection to TWS abruptly then client often can not reconnect using same client id.

One would need to restart TWS or wait for things to self-release on TWS app side (up to about 1 hour).


Code Examples

[Q] Where can I find C++ and C# tutorials/examples using ActiveX?

[A] by Rich Holowczak (a.k.a. “Prof. H.”) (src)

Folks

A while back a wrote a series of tutorials for the Interactive Brokers API for my Financial IT classes. I am now getting around to assembling them on my web site.  I have a few going up now using Visual C++ and Windows Forms. I will be posting more over the next few weeks that include WinForms with C# as well. Your comments and suggestions are welcome:

http://holowczak.com/category/ib/

Cheers,

Rich H.

[Q] Where can I find working posix C++ code examples?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A1] by Jim Covington

Sample code is on your machine inside the IB API installation folder.

[Q] Where can I find MORE ADVANCED working posix C++ examples?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] There is a great IB API Wrapper TwsApiC++ by Jan Boonen (see on GitHub)

(quote)

In the C++ implementation, the basic two API classes are EWrapper and EClient, in combination with the other classes/structures defined in the directory 'Shared'. Comparing with Java, these classes are pure interfaces. The implementation of EClient is done in  EClientSocketBaseImpl and at linking time, all is connected. The implementation of EWrapper is up to you.

Almost every C++ programmer opts to write the definition of a class and its implementation in two distinct files, although C++ does not enforce it. I prefer more the Java way, although C++ does not support it to the same extend. Some care is needed, and when pure interfaces are needed, the definition and implementation has to be separated anyway. So when studying different C++ projects, you will discover different approaches. Discussing all the pro's and contra's is a never ending story, as are the programming text style discussions.

The non-posix version of the api cannot be used as stand alone. It depends on the Windows specific socket implementations for reading the socket and calling the EWrapper methods. So the 'EReader' is somewhere down deep in the windows specific asynchronous socket stack.

This makes the non posix version non-portable and unusable for a non-windows program. To overcome these two weaknesses, some free libraries exist. I started the free TwsApiC++ library in 2008. The current version is based on the posix library of IB.

What does this TwsApiC++ library offers you:

a) the EClient and EWrapper are still the basic two API classes (or more specific, a derived version of it EClientL0 and EWrapperL0). So no need to access all the internal classes such as EClientSocketBase, or the other complex posix classes.

b) it implements an EReader, which is turned on by default. It runs in a separate thread, and waits for the incoming data on the socket, and calls the EWrapper methods without any delay. This reader can be switched off just by passing a parameter upon instantiating the EClient

c) it implements an empty method for every EWrapper method defined. So you only have to implement the one you are interested in

d) It protects the inner working of the IB library, where the TwsApiC++ is a wrapper around, from the possible exceptions thrown in the users code, but not catched by it.

e) It protects the EClientSocketBaseImpl from concurrent use of its internal members, which could lead to loss of requests send to the TWS

f) It ensures the data stuck in the buffer of EClientSocketBaseImpl  and failed to be send to TWS for some reason, is send as soon as possible, and not sits there waiting until the next EClient call is made. In the posix api, you the programmer has to take care of that(!)

g) optionally, it provides a way to check the correct spelling of the many textual or numeric parameters upon compile time instead of runtime.

The library is small and fast.  A few examples/test programs are included, and also project files for Visual Studio and make files for linux/unix are included

It includes a readme file.

See it on GitHub:

https://github.com/JanBoonen/TwsApiCpp

Or for group members:

See Files  > C++ Code > TwsApiC++ Directory

There are other implementations too, other members of this group know more about these.

Have fun with it.

Jan

(end of quote)

see also tutorial how to install TwsApiC++ here: [Q] Installing IB Gateway and TWS API for linux? (tutorial by Rich Holowczak (a.k.a. “Prof. H.”))

Note from Kurt:

when I *was* using TwsApiC++ I had to do extra work to *undo* the fact

that (like Java, I guess) TwsApiC++ creates a separate thread to read the

socket, and callback messages come in on that "reader thread". So I needed

another layer of buffering to get the callbacks to come in on the main

thread, and this required a tricky set of function overloads which pretty

much required modifying the TwsApiC++ implementation classes due to how

things were (not) factored.

The posix TWSAPI either does not require any such effort. Either there is

not a separate reader thread, or the buffering is already taken care of. (I

can't remember which.)

-Kurt

Another note from Jan came from here

JanB

Message 16 of 28 , Oct 20, 2010

View Source

Hi all,

I second this: the new improved API shouldn't reinvent the wheel, but rather make the current IB API a 'more round wheel and easier to turn' (hope this makes sense).

As Shane Cusson states, implementing a simple strategy with IB's API is 'like using a bazooka to swat flies'. So this new API should simplify its use, but should not prevent the implementation of complex strategies either.

For the C++ implementation, something like TwsApiC++ is a first step as it hides the use of the sockets. Not ESocketClient is the interface, but EClient/EWrapper is. Having to deal with the sockets in your ATS is fundamentally wrong, isn't it? TwsApiC++ more or less levels the C++ API with something like the API from dinosaurtech for C#.

On top of that, I'm implemented/ing two extra layers and I thought it could be interesting to share them here:

In a first layer on top of EClient/EWrapper (EClientL0/EWrapperL0):

1) The functionality of EClient/EWrapper is regrouped in functional classes such as Connection, MktData, HistoricalData, PlaceOrder, Account, Portfolio, etc. Each class keeps the data needed to make the request, and all EWrapper events related to the request are routed back to the requesting class. I.e. each reqHistoricalData to be executed needs its own instance of the class HistoricalData, which handles the request and receives all the historicalData() events for only his request.

2) It hides all the Id's (tickerId, orderId, reqId ): the allocation of the id's is automated and hidden. Errors with id==-1 are routed to the Connection class.

3) The classes for Account and Portfolio filters the events on the acctCode, so each account can have its own instance of Account and Portfolio. A blank/empty acctCode makes it to receive the data for all the accounts in case of.

4) Each of these classes have a common interface of a Request(), Cancel(), Resend() and Error(). The parameters for the req… are set during the allocation of the classes, so the Request() method has no parameters.

As the first layer resembles the EWrapper interface and does not adapt to any special behaviour of the IB API, the second layer goes one step further to make live easier for the programmer as follows:

1) it stores the data for classes such as Account, Portfolio, HistoricalData etc. (not for Mktdata or RealTimeData i.e. )

2) offers a more uniform interface when possible: OnInit() (called after the …End() events meaning all Account is available, Historicaldata is complete etc), OnChange()(i.e. after the Init() was called, and new updated Account data receives), and a ForEach(), forward and backward Iterators, etc. to access this data.

3) Split up of the MktData into MktDataSTK, MktDataFUT etc. The work-around for the "zero size bug" of Mike Smith does fit in this layer nicely.

4) a connection persistence and Resends the requests in case the link was broken with error (1101), 1102, 1300

5) A run method in the Connection class: i.e. Run(Run::Until(15,30,00)); which basically loops and calls checkMessages() and Connection::RunActions(). So my ATS or test programs have all a very compact 'main' routine: allocate the ATS and a call to the Run method.

I hope this view, although implemented in C++, is of any help. The implementation is not complete yet.

Jan Boonen

[Q] Python?

[A] For IB API + python awesomeness please check out:

The ib_insync package: https://github.com/erdewit/ib_insync

and related discussion group:

https://groups.io/g/insync

Also sarch groupse.io forum by word “jupyter”

https://groups.io/g/twsapi/search?q=jupyter

And then also:

https://groups.io/g/twsapi/message/37515

Use a GUI library like tkinter or pygame.  For testing you could also use jupyter,  there's a sample in here somewhere.

https://groups.io/g/twsapi/message/37329

https://groups.io/g/twsapi/message/37341

[Q] Delphi?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by josejp1

In this link are good examples of programming of Trader Workstation API with Delphi

http://delphimagic.blogspot.com/2010/07/interactive-brokers-con-delphi-tws.html

Ej. Connect with TWS, request market data, get ticks of price, place an order (mkt, lmt, stp, etc), request historical data, get information of an order, how to cancel orders

— also —

Delphi / BCB / Kylix TIABSocket component & API

http://www.hhssoftware.com/iabsocketapi/index.html

updated to the latest 9.76 API (January 2020)

This component set is built to work with the Interactive Brokers TradeWorkStation API. It allows Delphi and BCB programmers to build apps that send orders directly to the TWS. This TIABSocket component links directly to the socket to communicate with the TWS. There are no dll, ActiveX, .Net or DDE files required, and all socket messages & data processes instantly. This direct connection to the socket makes for a fast reliable interaction and response time. The component set also controls all interactions with the TWS, and includes sub components to manage order activity, portfolio and account details. The code looks and feels like any other Delphi component, and the syntax is similiar to existing standard component code. The complete source is included, along with a demo app and help file.

[Q] Java?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by Suminda

I have started a CEP (Complex Event Processing) based trading system which will be agile as possible. This is at a conceptualisation stage where we are running a few experiments on how the final architecture will be. Is it possible to help out in this as well as help build the team to take this forward.

Suminda

http://code.google.com/p/cep-trader/

java sidenote: by hubert.tse@gmail.com

My Eclipse was set on Java Compiler -> 1.5 Compliance Level.

Apparently 1.5 does NOT support @override and thus will not run. Once you set this to 1.6 and beyond, all the errors go away and it launches perfectly.

Many thanks for all your help!

[A] by Rob Terpilowsky (thread)

The 2 libraries I've used are

ta4j: https://github.com/mdeverdelhan/ta4j

and TA-Lib :http://www.ta-lib.org/

I prefer ta4j as it has a cleaner API than TA-Lib.

The 2 libraries I've used are

ta4j: https://github.com/mdeverdelhan/ta4j

and TA-Lib :http://www.ta-lib.org/

I prefer ta4j as it has a cleaner API than TA-Lib.

[Q] ruby?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by Dr. Hartmut Bischoff

Well, there is ib-ruby, too.

You can use pure ruby scripts, like

 require 'ib'

        ib = IB::Connection.new :port => 7496

        ib.subscribe(:Alert, :AccountValue) { |msg| puts msg.to_human }

        ib.send_message :RequestAccountData

        ib.wait_for :AccountDownloadEnd

        ib.subscribe(:OpenOrder) { |msg| puts "Placed: #{msg.order}!" }

        ib.subscribe(:ExecutionData) { |msg| puts "Filled: #{msg.execution}!" }

        contract = IB::Contract.new :symbol => 'WFC', :exchange => 'NYSE',

                                :currency => 'USD', :sec_type => :stock

        buy_order = IB::Order.new :total_quantity => 100, :limit_price => 21.00,

                                :action => :buy, :order_type => :limit

        ib.place_order buy_order, contract

        ib.wait_for :ExecutionData

https://github.com/ib-ruby/ib-ruby

Dr. Hartmut Bischoff

Stuttgart

Update 06-Jun-2018: by Dr. Hartmut Bischoff:

View/Reply Online (#39948) 

(quote)

Imagine you got a trading robot at a remote location, accessible only via an reverse ssh tunnel, or a TWS is running on a cheap cloud-server.

Simple-Monitor ( https://github.com/ib-ruby/simple-monitor ) is designed to display the status of a Friends&Family-Account via web-interface.

It comes with just 435 lines of code using the camping micro-framework and ib-ruby.

Its intended to run even on resource-low  hardware and serves well running elinks in a tmux-session, avoiding a running TWS-GUI in favor of the TWS-Gateway

It too demonstrates how well ib-ruby ( https://github.com/ib-ruby/ib-ruby ) integrates in existing frameworks.

(end of quote)

[A2] by Peter Gum

IBCsharp, tradecommander, openquant,

tradelink and others. RighEdge <http://www.rightedgesystems.com/> is as

comprehensive as any, including supporting VB.NET

google "jTWScharts"

[Q] C#?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A1]

see: Interactive Brokers C# Api (by Karl Schulze)

ib-csharp

https://github.com/krs43/ib-csharp

and/or:

http://www.dinosaurtech.com/utilities/

[A2] by Rich Holowczak (a.k.a. “Prof. H.”)

I have written a few more to work with IB's C# API (console only applications). same location:

http://holowczak.com/category/ib/

Cheers

Rich H.

if you aware of any other published working code examples, please let me know so I can add the link into this FAQ

[Q] R

[A] by ce (from this thread)

pdf files in

http://cran.r-project.org/web/packages/IBrokers/index.html

are good.

 

Also there is a very good example system in

http://censix.com/download/

[A2] by censixtws (from this thread)

Hi I have developed this collection of R scripts that I call qsiblive ( downloadable from my website http://censix.com ) and am currently testing it with my paper account. It trades one symbol on an EOD basis. I think its strength lies in the immense flexibility that it provides with regard to 'plugging-in' R analytics packages/methods (some classic indicators, some ARMA, GARCH modeling, you name it). Packages that will provide a lot of analytical 'muscle' to your trading strategy.

I am making this available to get some feedback from the community of seasoned traders/developers here.

Cheers

Soren

see:

censix – pure R intraday trading framework – screencast01

https://www.youtube.com/watch?v=Zzd7tx452Ao&index=2&list=LLYax4ZgrNYHP5fLotKBc9Lg

censix on GitHub: https://github.com/censix/INTRADAY-PartAB

[R-related resources]

Contemporary Portfolio Optimization Modeling with R

https://www.youtube.com/watch?v=hKDalfhDawA

[Q] Pattern recognition library?

Added on 16-May-2017

is there an open source library for chart patterns? or at least algorithm so i can use to program it myself?

looking for flag, triangles and double top/bottom mostly.

[A] by Johnny Red (View/Reply Online (#37504))

I use TA-Lib – it's open source and very good.  It's written in C++ and is available for Java/C#/etc.  I even think there is a version for Python.

Good stuff.

http://ta-lib.org/hdr_dw.html

[A2] by Peter Cawthron

There’s a Python wrapper for TA-Lib at https://github.com/mrjbq7/ta-lib

Regards,

Peter

[Q] Reverse engineering java?

[A]

I know you mentioned you've resolved your issue, but just FYI, if you want

to peek at the contents of "other.jar":

http://java.decompiler.free.fr/

Grab the GUI version and click "File->Open", pick "other.jar" and browse the

source. The functionality is essentially the same as the .Net Reflector.

–s

ps. if reverse engineering is illegal in your country, you didn't hear this

from me. (I'm Canadian, things are pretty lax up here.)

——- also ——–

Alternatively, make a copy of it and rename to "other.zip" and open as

usual with Windows or your Mac.


Implementation notes

[Q] What value means "no value"?

[A] by Kurt (src)

IB shows no consistency in the use of values meaning "no value".  Sometimes it is INT_MAX or DOUBLE_MAX, sometimes 0, sometimes –1, sometimes –0.99, etc.

The historical bar volume field is only populated for TRADES, never for BID, ASK, MIDPOINT, BID_ASK, OPTION_IMPLIED_VOLATILITY, or HISTORICAL_VOLATILITY.  Instruments that do not provide TRADES (such as FX) therefore do not have volume available.

There are probably many other things omitted for various instruments in ways that follow no particular pattern, and are not documented.  You just get what you get.  Don't worry about things looking strange.  That is the least of the problems you will have at IB.

I've definitely noticed that in historical data the unused fields in a bar are sometimes 0 and sometimes –1.  There is definitely no documentation about that.

–Kurt

[Q] Best practice to handle ID’s / nextValidId?

[A] by edbar@exmsft. from this thread

I store the ID on my hard drive.

Then when I start up TWS, I check the nextID that they send to see which one

is bigger.

If my stored ID is greater than the TWS number then I go with mine.

Then, every time I increment the ID number, I update my hard drive backup.

You can always go to the TWS CONFIG API screen and reset the api id numbers.

–Ed

[A2] by btw12342001 (same thread)

I don't think you should use the orderId for tickerId. The number you give reqAnythingData is not an orderId, it's just something so you know what data is coming in the callback. It's possible you subscribe to more than one symbol of data. It still doesn't explain why you're mixed up with orderId's

I guess I don't need to answer the 2002 question πŸ™‚

[A3] by Kurt (same thread)

A lot of people have reported here that just using nextValidId() is not

sufficient. I've been doing it without problems so I may not recall the

details, but I believe the common practice it to keep track (on disk) of

your greatest order id, and use the greater of that and the value returned

by nextValidId when you start up, and increment from there. You can check

the archives. This question was last asked less than 2 months ago I

believe.

Regarding reqMktData etc each one has its own id space but for your own

sanity you probably want to keep them unique. I keep all request id's

unique *except* for placeOrder for which I used a separate scheme. Using

unique request ids facilitates routing incoming request errors.

-Kurt

[Q] After connection lost I need a list of all open orders and positions in order to get synchronized?

[A] by src

Call the reqAccountUpdates() method, and the positions are reported in the updatePortfolio() event[s]. The events fire once for each position. Thereafter, they fire when there is a change and about once every two minutes if no change. It works well.

[rwk]

You might consider also requesting executions. Sometimes open orders or

execution reports (or both) may fail to work right, due to IB misfeatures or

whatever. The more information the better, so that means requesting

positions is also a good idea. Positions are part of account updates.

-Kurt

> What do I send in the acctCode parameter of reqAccountUpdates()?

You can set it to the null string if you do not have an FA account scenario.

-Kurt

or you can set it to: your account code UXXXXXX or FXXXXXX

> What do I receive in the position parameter of updatePortfolio()? Is it the

number of contracts in the Portfolio

YES

[Q] Tracking time (internally)

[A] by Dmitry

This has nothing to do with any incoming timestamps from outside – it is just for local timestamping of all internal events. As the good enough resolution I choose millisecond resolution. Local time is kept my NTP with updates polled from ntp servers as often as we can (had to build own Stratum-1 ntp server based on gps pps signal (see Time Drift for more details))

long get_epoch_ms()

{

    // http://brian.pontarelli.com/2009/01/05/getting-the-current-system-time-in-milliseconds-with-c/

    timeval t;

    gettimeofday(&t, NULL);

    long epoch_ms = (t.tv_sec * 1000) + (t.tv_usec / 1000);

    //std::cout << ";" << t.tv_sec << ";" << t.tv_usec << " or simply epoch_ms: " << epoch_ms << std::endl;

    return epoch_ms;

}

[Q] Building live Trades

see also relevant: [Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] What is the sequence of tickSize() tickPrice events?

[A] by Dmitry

If you want to extract events for closed deals (not some BID/ASK moves noise, but really closed deals with real price/volume), then you gotta follow these considerations since it might be not quite straight forward due to IB would only send relative “differences” instead of absolute self-contained discrete messages about trades. To warm up, please read: [Q] Sometimes lastSize=0. wtf?

Now the implementation algo to get trades Dmitry uses:

– have container for  trades within MyContract class

– all tickPrice(), tickSize() events are routed by ewrapper to request object (RequestMktData), which is associated with MyContract class instance, which in turn has tickPrice() tickSize() methods to process incoming ticks

– now inside MyContract class:

void MyContract::tickPrice(Account *account, long epoch_ms, TickerId tickerId, TickType tickType, double price, int canAutoExecute) {

    LOG(INFO) << "MyContract::tickPrice";

    // forward raw tick to subscribed algos (if any)

    for (std::vector<AlgoBase*>::iterator ialgo = subscribed_algos.begin(); ialgo != subscribed_algos.end(); ialgo++) {

        (*ialgo)->tickPrice(account, this, epoch_ms, tickerId, tickType, price);

    }

    // analyze if GotTrade()

    if (tickType == LAST) {

        // we got new trade, but we have to wait for TickSize() call to

        // complete the Trade obj (and then will call GotTrade(trade) to pass

        // complete Trade object to all subscribers)

        this->tradesPerAccountMap[account_as_key].push_back(new Trade(account, epoch_ms, tickerId, price));

    }

. . .

}

void MyContract::tickSize(Account *account, long epoch_ms, TickerId tickerId, TickType tickType, int size) {

    LOG(INFO) << "MyContract::tickPrice";

   

    // forward raw tick to all subscribers

    for (std::vector<AlgoBase*>::iterator ialgo = subscribed_algos.begin(); ialgo != subscribed_algos.end(); ialgo++) {

        (*ialgo)->tickSize(account, this, epoch_ms, tickerId, tickType, size); // forward tickSize() event to subscribed algos (if any)

    }

        // analyze if GotTrade()

    if (tickType == LAST_SIZE) {

. . .

        // CASE 1 of 3: check if last_trade is in "waiting for size" state (last_trade->size < 0 aka 'uninitialized'))

        if (last_trade->size < 0) {

            // last_trade's size is not yet initialized, means we waited for this tickSize() event

            // to complete the trade obj (and forward it to all subscribers)

            last_trade->size = size;

            last_trade->completion_time_ms = epoch_ms – last_trade->epoch_ms; // we track how long it took us to complete this trade object

            last_trade->is_complete = true;

            // forward complete trade to all subscribers

            gotTrade(last_trade);

            return;

        }

        // CASE 2 of 3: Is it duplicate of last_size tick (same size as last complete trade and within 9ms) => just ignore it

        if (last_trade->expect_dup_ticksize && last_trade->size == size && (epoch_ms – last_trade->epoch_ms) <= 9) {

            last_trade->expect_dup_ticksize = false; // mark last_trade as "got expected duplicate" so next tickSize if

            // happen to arrive within <7-8ms with the same size field would not be treated as duplicate anymore

            // (the 'expect_dup_ticksize' flag allows us to ignore tickSize only once

            // and only for expected ticks (ones that go after "Last" 2 times in a row))

            return;

        }

        // case 3 of 3: It is normal LAST_SIZE tick telling us about new trade with the same price as before! We got new trade!

        Trade * new_trade = new Trade(account, epoch_ms, tickerId, last_trade->price, size, true, 0, false);

        ptrades->push_back(new_trade);

        // forward complete trade to all subscribers

        gotTrade(new_trade);

        return;

    }

}

[Q] Building live bars

[A]

https://groups.io/g/twsapi/topic/4046604

https://groups.io/g/twsapi/topic/4046154

[Q] Why do we get two identical tickSize() callbacks?

[A] explained by Richard L King

Date: Sat, 5 Feb 2011 00:47:45 -0000

Subject: RE: [TWS API] Re: tickSize(): LAST_SIZE and VOLUME order

Perhaps I can throw a little light on this subject to explain what is

actually happening here and why.

In the early days of the API the TICK_PRICE protocol message (ie the message

itself, not the callback function/event) had a version number of 1 and had a

price field, but no size field.

So at that time, all prices came in TICK_PRICE messages and all sizes came

in TICK_SIZE messages. Bear in mind the sampling mechanism that IB uses,

which only sends values that have changed during the previous sampling

interval. If a price had changed, it would send a TICK_PRICE message before

the corresponding TICK_SIZE message. But this approach had a problem,

because a TICK_SIZE message was only sent if the size had changed. Thus

there was no way of the client knowing, when a TICK_PRICE message arrived,

whether the corresponding size was the same as the previous size (in which

case there would be no TICK_SIZE message) or whether it too had changed and

a TICK_SIZE message would follow.

To solve this problem, IB introduced version 2 of the TICK_PRICE message in

API v760 Beta (my copy of this is dated Feb15 2004). This included a size

field, and now whenever a TICK_PRICE field was sent (which meant a change of

price during the sampling interval had changed), the size could be included

whether or not the size itself had changed. The TICK_PRICE message would now

generate both a tickPrice() callback followed by a tickSize callback. This

meant that clients had it nice and easy: when you got a tickPrice() callback

you just had to remember the price, and any following tickSize()s would

implicitly relate to that price until the next tickPrice().

However there was now a different problem. Any client that was using an

older version of the API with client version less than 15 would not

understand the version 2 message and so TWS would continue to send version 1

TICK_PRICE messages to those clients. But this meant that the tickSize()

callback following the tickPrice() callback would not be generated, so the

client would not have the full information. To fix this, after sending the

TICK_PRICE message, IB decided to also send a TICK_SIZE message with the

same size as the TICK_PRICE message. Thus clients with version less than 15

would now receive the full information.

The stupid thing was that IB should have only sent the extra TICK_SIZE where

the client version was less than 15, but they actually sent it to all

clients, and of course they still do. So now we have this silly situation

where after every TICK_PRICE message a redundant TICK_SIZE message is sent,

resulting in a tickPrice() callback followed immediately by two identical

tickSize() callbacks.

So why don't IB fix this? Well, I don't really know. It may be they think

it's of no significance, because for example anyone watching the data

onscreen would never notice. It's only because we like to tally things up

and compare with the cumulative volume figures that it irks us (and of

course because it's just plain stupid!).

And once you understand what's going on it's easy enough to code round it,

as Mike has already pointed out below.

If anyone wants to complain to IB about this and persuade them to fix it, by

all means have a go, but I very much doubt whether they'd show any interest

at all. They haven't for the past 7 years (and after all they themselves

must have had to code round it in TWS).

Actually the real annoyance here is not so much the issue itself, as the

fact that so many people spend so much time discovering it, worrying about

it, and inventing workarounds. But then that's the case for so many aspects

of the API…

Richard

[A] explained even more by Richard L King (found here)

Added on 09-Jun-2015

Let’s summarise what’s going on here:

1. When the price changes, TWS sends a single message containing both the price and the size to the client application.

2. IB’s API code then makes separate tickPrice and tickSize callbacks (note that the latter is immediately after the former – you can easily see this in the API code.

3. IB’s servers then send the size again, and the API makes another identical tickSize callback (the reasons for this are explained in my contribution to the FAQ linked to by rholowczak’s post on 2 June – this is the part highlighted in blue: I suggest you read that carefully).

4. Each time there is a trade with the *same* price but a *different* size, another tickSize callback is made.

5. When there is a trade with the *same* price and the *same* size, no tickSize callback is made.

From this you can immediately deduce that there will be *at least* 2 tickSize callbacks per tickPrice callback, and usually more because the price doesn’t change with every trade.

By the way, there is no need to fiddle around with ‘blanking time’ that you mentioned in your previous post. If you get a tickPrice callback, just record the price. The very next callback is *always* the tickSize (check the API code to convince yourself that this is true), so record the size and call your application code to process the combined price/size. Subsequently whenever you get a tickSize callback, if the size is the same as the last size you recorded, just ignore it (because it will be the duplicate tickSize described in 3 above), otherwise call your application code to process the combined price/size where the price is that recorded in the latest tickPrice callback.

It may be worth pointing out that in spite of what I said in 5. above, there *are* occasions where a tickSize occurs where the size is the same as the previous one and the price has not changed – however they are rare, and given the approximate nature of the data feed it’s not worth trying to complicate things to try and detect them. Having said, that my own API code does complicate things somewhat, but I’m not sure what I gain by doing so. So I won’t bother to describe these extra wrinkles unless someone is really interested.

see also somewhat related:  [Q] Building live Trades

[Q] Could someone suggest a good algorithm for receiving market data for more than 100 symbols with a single account?

[A] by Jason Sapp

I use a "round robin" type of algorithm that utilizes one of my paper trading

accounts to obtain cumulative volume on about 1000 symbols. I am not too

concerned if my cumulative volume number is off by a couple minutes, so this

works for me. The way I do it is as follows:

-Call reqMktData (which contains cumulative volume) for the first 90 symbols in

my list (making sure not to call reqMktData more than 30 times per second—at

least I think 30 was the magic number, but this limitation may longer be in

place).

-Wait 10 seconds (after calling reqMktData on my the last of the 90 symbols).

-Call cancelMktData on all of those symbols.

-Obtain the next 90 symbols from my list.

-Start over.

The reason I used 90 symbols was because I sometimes had a market row or two

being used by my paper trader TWS.

Jason

————————- also alternative ———————————

from here

I use Yahoo Financial's CSV feed b/c I can request multiple symbols within a

single request to get quick price data snapshots for each symbol. For

example, to request last price, daily high, daily low, close, and volume for

2000 symbols, it takes like 10 seconds to get that information from Yahoo.

I use this specifically in the morning when scanning through stocks. This

is more efficient, quicker, and reliable than trying to round-robin my way

through TWS requests and dealing with the request limits they have on their

accounts.

…   by Brant Hahn

[Q] data links drop; feeds fade. Serious software needs to handle that. How?

[A] implementation idea by Russ Herrold

As a practical matter, by subscribing to some indicator that

is continuously updated as well as a more leisurely symbol,

one always has some value appear during that second when the

link is live — watch ES or such πŸ˜‰

[Q] Request/response/error architecture thoughts?

[A] by Kurt (from this thread: C++ architecture question)

There are hundreds of ways of doing it.  I don't believe in "ideal".  I'm posting some code here because it might be useful for some others as well as you.  (From this code you could probably reproduce a big chunk of the request/response/error architecture I use.)

I implemented request tracking classes with an instance per request.  The default behavior is for the constructor to send the request but it is also possible to create the request and defer sending it.  The class provides a bunch of useful functions including tracking latency, logging, and matching responses and errors to requests.

For this purpose I have an ActiveRequest class with subclass ActiveRequestWithContract for contract-based classes, adding additional information to the generic logging capabilities provided by ActiveRequest.  Then I further have a separate subclass for each IB request type which is either a subclass of ActiveRequest or of ActiveRequestWithContract.  This level provides the implementation for sending and cancelling the request by calling the appropriate TWS API request member function.

I use further subclassing for each distinct purpose or for distinct interfacing requirements, such as dispatching into Objective-C handlers.

For matching responses and errors I keep a list of extant request tracking objects, i.e. those objects that have not been deleted, which is an independent consideration from whether they have been sent, have received a response or a full set of responses, etc.  The ultimate client of the request can determine whether the request object should be deleted when the response is complete, if applicable.

As a result there is a list of existing requests which can be searched by request id.  This permits incoming responses and incoming errors that reference request id's to be routed to the appropriate request tracking object.  Thus request clients can be notified when a request is aborted due to an error.

This is not optimal, so perhaps deficient and certainly not ideal.  If performance were an issue the list of requests could instead have been a B-tree, or even a hash table.  So you could use the associative arrays provided by the C++ STL.

In any case the bulk of request tracking implementation is quite removed from the tickPrice callback.  However for each request tracking subclass that is specific to a particular IB request type, I have a method (member function) with name and parameters identical to the corresponding EWrapper member.  I just copy/paste the function prototype.  Thus a part of the class hierarchy looks like this:

    MarketDataRequest : ActiveRequestWithContract : ActiveRequest

and while my EWrapper subclass of course has this member

   virtual void tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute);

my MarketDataRequest class also has

    virtual void tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute) = 0;

for which the implementation is left to the next-level subclass.  This approach gives me a generic interface to the TWS API with none of the specifics of my app present in it.

The generic tickPrice implementation then does a linear search of a doubly-linked list to find the request object to route to, and looks like this.  The superclass of MyWrapper provides a default implementation that just logs the request.  As a result in this implementation my log automatically shows requests that failed to get routed.

void MyWrapper::tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute)

{

    for (ActiveRequest *req = marketDataReqTracker.Next(); req != &marketDataReqTracker; req = req->Next())

    {

        if (req ->ReqId() == tickerId)

        {

            if (MarketDataRequest *specificReq = dynamic_cast<MarketDataRequest *> (req))

            {

                specificReq ->tickPrice (tickerId, field, price, canAutoExecute);

                return;

            }

        }

    }

   

    inherited::tickPrice (tickerId, field, price, canAutoExecute);  // default logging used when response not routed above

}

It is not perfectly efficient, but it is "perfectly adequate" for my purposes and easy enough to upgrade if needed even though I am not using container abstractions (STL or otherwise).

The use of dynamic_cast is strictly unnecessary, because marketDataReqTracker should reliably contain only MarketDataRequest objects.  So this code could have benefited from some template use.  But even without that a static_cast would have been safe and only because the performance hit is no issue I just did the conservative thing.  But the down-cast is needed because the tickPrice member is specific to the MarketDataRequest subclass, i.e. the entire EWrapper interface is definitely not contained in each request object, per the description above.

Meanwhile error handling somewhat parallels this, with my EWrapper subclass providing this implementation

void MyWrapper::error(const int id, const int errorCode, const CString errorString)

{

    (void) error_Route (id, errorCode, errorString);

}

and error_Route special-cases a bunch of error codes for more sensible and direct routing and otherwise calls ReqTracker_error_Route:

ActiveRequest *MyWrapper::error_Route(const int id, const int errorCode, const CString& errorString)

{

    if (errorCode == 505)

        // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

        printf ("generic error %d %s [id=%d]\n", errorCode, errorString.CStr(), id);

    else if (id == kIBNullReqId)

        printf ("generic error %d %s\n", errorCode, errorString.CStr());

    else

        printf ("request %d error %d %s\n", id, errorCode, errorString.CStr());

   

    bool handled = false;

    ActiveRequest *handledBy = NULL;

   

    switch (errorCode)

    {

        // errors specific to Orders

        case 103:    // [from doc]  Duplicate order ID.

        case 104:    // [from doc]  Can't modify a filled order.

        case 105:    // [from doc]  request 45 error 105 Order being modified does not match original order

        case 106:    // [from doc]  Can't transmit order ID:

       

        …

        case 121:    // [from doc]  Invalid BD flag for the order. Check "Destination" and "BD" flag.

        case 122:    // [from doc]  No request tag has been found for order:

        // 123, 124 vague

        case 125:    // [from doc]  Buy price must be the same as the best asking price.

        case 126:    // [from doc]  Sell price must be the same as the best bidding price.

        case 129:    // [from doc]  VWAP orders must be submitted at least three minutes before the start time.

        case 131:    // [from doc]  The sweep-to-fill flag and display size are only valid for US stocks routed through SMART, and will be ignored.

        case 132:    // [from doc]  This order cannot be transmitted without a clearing account.

        case 133:    // [from doc]  Submit new order failed.

        case 134:    // [from doc]  Modify order failed.

        case 135:    // [from doc]  Can't find order with ID =

        case 136:    // [from doc]  This order cannot be cancelled.

        case 137:    // [from doc]  VWAP orders can only be cancelled up to three minutes before the start time.

        // 138, 129 vague

        case 140:    // [from doc]  The size value should be an integer:

        case 141:    // [from doc]  The price value should be a double:

        case 142:    // [from doc]  Institutional customer account does not have account info

        // 143 vague

        case 144:    // [from doc]  Order size does not match total share allocation.  To adjust the share allocation, right-click on the order and select “Modify > Share Allocation.”

        // 145 vague

        case 146:    // [from doc]  Invalid trigger method.

        …

        case 2102:    // [from doc]  Unable to modify this order as it is still being processed.

        case 2109:    // [from doc]  Order Event Warning: Attribute “Outside Regular Trading Hours” is ignored based on the order type and destination. PlaceOrder is now processed.

                    // actual:     Order Event Warning:Attempted modify of OutsideRth will not be done.  Order modify now being processed.

       

            // All the errors listed above are thought to be specific to Orders based on the message description.

            // It is sometimes unclear from the message whether the order id will be available with the error, or not.

            // Only errors with an id can be meaningfully routed to Order_error_Route.

            // All the rest might as well be routed right away to ReqTracker_error_Route.

            //

            // In any case the above list of errorCode's are treated as a list of what should be preferentially routed to orders before request trackers.

            // However the consequences of including an extraneous errorCode in the above list are non-existent for errors that turn out not to include a request id.

            // For errors with a request id, an extraneous errorCode in the above list is consequential only if it is NOT an order error,

            // but the request id coincidentally matches a currently-tracked order id.

            // It is hard to comment meaningfully on how obscure that is.

            // Instead it suggests motivation for a redesign that keeps order id's in their own numeric space,

            // or interleaves them in the same numeric space with other request id's.

            … rest of long comment omitted

            if (id != kIBNullReqId)

                handled = Order_error_Route (id, errorCode, errorString);

            else//10/14/12 was:  if (!handled)

                // So both kIBNullReqId errors and errors that are not successfully routed to an order based on the id will end up here.

                handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                    // see comment in default case re ReqTracker_error_Route details

            break;

            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

           

        case 326:    // error -1 326 Unable connect as the client id is already in use. Retry with a unique client id.

            clientIdInUse = true;

            break;

       

        case 505:    // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

            // Do not route this message–there should be no use in doing so since the connection is about to be closed.

            // Also see special case at top of this function.

            break;

       

        case 2100:    // actual:       New account data requested. API client has been unsubscribed from account data.

            handledBy = accountUpdateReqTracker .error_Route (id, errorCode, errorString);  // make sure AccountUpdateRequest::error_Generic gets this error

            AccountUpdatesCancelledByServer (errorString);

            break;

       

        default:

            handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                // calls ActiveRequestCategoryManager::error_Route which calls ActiveRequest::error_Route for each outstanding request

                // which calls error_SpecificRequest for request with matching id if found

                // and may call error_Generic which is currently only implemented by AccountUpdateRequest

                // for error 2100 "New account data requested. API client has been unsubscribed from account data."

           

            if (handledBy == NULL)

                if (id != kIBNullReqId)

                    (void) Order_error_Route (id, errorCode, errorString);

            break;

    }

   

    return handledBy;

}

and otherwise calls ReqTracker_error_Route which is also less than ideal but perfectly adequate and does this.

ActiveRequest *MyWrapper::ReqTracker_error_Route (const int id, const int errorCode, const CString& errorString)

{

    if (ActiveRequest *routedTo = historicalDataReqTracker  .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = marketDataReqTracker      .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = contractDetailsReqTracker .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = realTimeBarReqTracker     .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = accountUpdateReqTracker   .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = marketDepthReqTracker     .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = fundamentalDataReqTracker .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = scannerReqTracker         .error_Route (id, errorCode, errorString))

        return routedTo;

    return NULL;

}

which in turn calls this

ActiveRequest *ActiveRequestCategoryManager::error_Route(const int id, const int errorCode, const CString& errorString)

{

    // Route all errors to all objects, until/unless an object's error_Route returns non-NULL, indicating it has "handled" the error.

    // In a typical implementation, generic errors (id == -1) will end up being routed to all objects,

    // and specific errors will be ignored except by the object with a matching request id (if any),

    // which will "handle" the error and return non-NULL (usually pointer to itself), preventing further looping here

    // (and in current implementation preventing equivalent of further looping in MyWrapper::ReqTracker_error_Route which calls this function for each request category until it is "handled").

    for (ActiveRequest *req = Next(); req != this; req = req->Next())

    {

        if (ActiveRequest *routedTo = req ->error_Route (id, errorCode, errorString))

            return routedTo;  // very likely routedTo will always equal req

    }

   

    return NULL;

}

and there the generic implementation of error_Route comes into the picture

ActiveRequest *ActiveRequest::error_Route(const int id, const int errorCode, const CString& errorString)

{

    if (this->ReqId() == id)

    {

        error_SpecificRequest (id, errorCode, errorString);

        return this;

    }

   

    if (id == kIBNullReqId)

    {

        if (errorCode == nonAPIErrorCode_connectionClosed)

            Aborted (errorCode, errorString);

        else

            error_Generic (errorCode, errorString);  // as of 5/3/12 this is the only call to error_Generic

    }

   

    return NULL;

}

And an implementation of error_SpecificRequest is provided by the request-tracking subclass for each API request type, such as this one.

void MarketDataRequest::error_SpecificRequest (const int id, const int errorCode, const CString& errorString)

{

    if (errorCode == 200)

    {

        Aborted (errorCode, errorString);

    }

   

    /* market data error reference

   

        error 39 200 No security definition has been found for the request

            the same error can happen for reqMktData and reqContractDetails and reqHistoricalData

            in this case it was a bogus combo (spread between AAPL and INTC options)

                sending MarketDataRequest 39 AAPL,INTC SPREAD

            but the "No security definition" error should be dispatched to the final request object

            and should maybe delete without cancel

           

        are there existing cases in which [??? text lost ???] caused an unintended cancel?

    */

}

So that may give you some sense for an approach.  While not ideal there is a lot of architecture there that supports quite a bit of experience working with the API.

-Kurt

[Q] Thoughts on “duplicate requestID” value (bad value given by API)

more details:

In my program, I get the NextValidId on connection to TWS and I use this

number to place my first order for the day, and I keep incrementing to place

the orders following it.

It so happened yesterday that the NextValidId returned by ID was a duplicate

one. The program had placed an order with order ID 638 few days back and It

placed the another order with the ID, 638.

(both the orders got executed)

I have unchecked the option "Reuse rejected order" in TWS — orders

configuration menu.

Does anyone have a clue why was an used OrderID returned by IB?

Thanks,

Vishruth

[A] by Frank Bell

I found it necessary to keep my own persistent copy of the current order

number. At start up I use max(my number, NextValidId).

Frank

[A] by Richard King

TWS stores the next valid id in its settings file. This is a large XML file,

and I suspect it's only written out when you shut down TWS (though I think

there is a save settings command in the menu? – can't check this at the

moment).

So if TWS crashed or you killed it rather than tidily closing it down, that

might account for your observation.

Richard

[A] by Jan Boonen

I also check the orderId of the EWrapper::openOrder( OrderId orderId, …) events, just in case of. If that order id is equal or bigger then the one to be used next, I set the next available value to orderId+1.

Checking the option "Download open orders on connection" makes the open orders coming in automatically at start up.

janB

[Q] Thoughts on “requestID space partitioned”

Hi Kurt,

Thank you for shading more light on the architectural constructs you developed.

Most definitely I'll reuse your approach of having one tracking instance for each individual request along with searchable lists (or maps as you suggested) to quickly retrieve corresponding objects for proper response routing/handling.

I was developing something very similar couple weeks before and part of the logic was automatic request ID generation at which point I recall that few months ago I read your take on request ID from some of the tread I can no longer easily find. If I recall correctly you said you track order request IDs separately from all other kinds of requests.

If it is the case, then my question is: how would you distinguish between "order's request IDs" space and "other request IDs" space if they overlapping? Example: you got error() call with ID matching one order tracking set and one from "other requests" set. Then route became ambiguous.

When I further thought about this problem I thought it would be generally good idea to add "expiration" field to each request, which can be set to something non-zero for requests with finite lifetime expectancy or to zero value for non-expirable ones (like for data subscription – should live "forever" until canceled). Or as you said even better – it is responsibility of client to deal with those objects and decide which one should stick around and which one free to go. But even hypothetical short life cycle of request tracking instances does not 100% complete less often "ambiguous route" cases if we try to separate orders from others. (and why do we want to do this in the 1st place).

ps: other possibility (if we need to separate orders from others by ID) would be to use odd numbers for orders and even for other requests, circling those spaces (like process ID assigned in linux OS:), but its ugly and unnecessary shrinks space for IDs twice (and I doubt we need same power of sets for orders vs other requests)

Thank you,

Dmitry

[A] by Kurt (from this thread: C++ architecture question)

[snip]

 

I was developing something very similar couple weeks before and part of the logic was automatic request ID generation at which point I recall that few months ago I read your take on request ID from some of the tread I can no longer easily find. If I recall correctly you said you track order request IDs separately from all other kinds of requests.

If it is the case, then my question is: how would you distinguish between "order's request IDs" space and "other request IDs" space if they overlapping? Example: you got error() call with ID matching one order tracking set and one from "other requests" set. Then route became ambiguous.

Mind you just because I have so far gotten away with this approach doesn't mean it is the best or that I will continue to get away with it.  I like to action on issues that are not pressing so I can keep considering different angles, or … maybe leave it as indefinitely.  Others have partitioned the requestID space.  I think Richard has done this.  It has been pointed out that it is a plentifully big space.

I just happen to like small numbers so I use small numbers for order IDs and small numbers for other request IDs.  Your odd/even idea (below) would also take care of this although I have the memory it may cause problems for certain kinds of orders for which IB may automatically insert another orderID on your behalf at the next integer value.

The reason I get away with it is that the error code space is somewhat defined by IB.  The MyWrapper::error_Route function which I mentioned but did not show has, among other things, a big switch on errorCode and a long list of errors known to be particular to orders, based on the IB documentation.  Here is that function with the long list of order-related errors truncated.

ActiveRequest *MyWrapper::error_Route(const int id, const int errorCode, const CString& errorString)

{

    if (errorCode == 505)

        // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

        printf ("generic error %d %s [id=%d]\n", errorCode, errorString.CStr(), id);

    else if (id == kIBNullReqId)

        printf ("generic error %d %s\n", errorCode, errorString.CStr());

    else

        printf ("request %d error %d %s\n", id, errorCode, errorString.CStr());

   

    bool handled = false;

    ActiveRequest *handledBy = NULL;

   

    switch (errorCode)

    {

        // errors specific to Orders

        case 103:    // [from doc]  Duplicate order ID.

        case 104:    // [from doc]  Can't modify a filled order.

        case 105:    // [from doc]  request 45 error 105 Order being modified does not match original order

        case 106:    // [from doc]  Can't transmit order ID:

       

        …

        case 121:    // [from doc]  Invalid BD flag for the order. Check "Destination" and "BD" flag.

        case 122:    // [from doc]  No request tag has been found for order:

        // 123, 124 vague

        case 125:    // [from doc]  Buy price must be the same as the best asking price.

        case 126:    // [from doc]  Sell price must be the same as the best bidding price.

        case 129:    // [from doc]  VWAP orders must be submitted at least three minutes before the start time.

        case 131:    // [from doc]  The sweep-to-fill flag and display size are only valid for US stocks routed through SMART, and will be ignored.

        case 132:    // [from doc]  This order cannot be transmitted without a clearing account.

        case 133:    // [from doc]  Submit new order failed.

        case 134:    // [from doc]  Modify order failed.

        case 135:    // [from doc]  Can't find order with ID =

        case 136:    // [from doc]  This order cannot be cancelled.

        case 137:    // [from doc]  VWAP orders can only be cancelled up to three minutes before the start time.

        // 138, 129 vague

        case 140:    // [from doc]  The size value should be an integer:

        case 141:    // [from doc]  The price value should be a double:

        case 142:    // [from doc]  Institutional customer account does not have account info

        // 143 vague

        case 144:    // [from doc]  Order size does not match total share allocation.  To adjust the share allocation, right-click on the order and select “Modify > Share Allocation.”

        // 145 vague

        case 146:    // [from doc]  Invalid trigger method.

        …

        case 2102:    // [from doc]  Unable to modify this order as it is still being processed.

        case 2109:    // [from doc]  Order Event Warning: Attribute “Outside Regular Trading Hours” is ignored based on the order type and destination. PlaceOrder is now processed.

                    // actual:     Order Event Warning:Attempted modify of OutsideRth will not be done.  Order modify now being processed.

       

            // All the errors listed above are thought to be specific to Orders based on the message description.

            // It is sometimes unclear from the message whether the order id will be available with the error, or not.

            // Only errors with an id can be meaningfully routed to Order_error_Route.

            // All the rest might as well be routed right away to ReqTracker_error_Route.

            //

            // In any case the above list of errorCode's are treated as a list of what should be preferentially routed to orders before request trackers.

            // However the consequences of including an extraneous errorCode in the above list are non-existent for errors that turn out not to include a request id.

            // For errors with a request id, an extraneous errorCode in the above list is consequential only if it is NOT an order error,

            // but the request id coincidentally matches a currently-tracked order id.

            // It is hard to comment meaningfully on how obscure that is.

            // Instead it suggests motivation for a redesign that keeps order id's in their own numeric space,

            // or interleaves them in the same numeric space with other request id's.

            … rest of long comment omitted

            if (id != kIBNullReqId)

                handled = Order_error_Route (id, errorCode, errorString);

            else//10/14/12 was:  if (!handled)

                // So both kIBNullReqId errors and errors that are not successfully routed to an order based on the id will end up here.

                handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                    // see comment in default case re ReqTracker_error_Route details

            break;

            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

           

        case 326:    // error -1 326 Unable connect as the client id is already in use. Retry with a unique client id.

            clientIdInUse = true;

            break;

       

        case 505:    // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

            // Do not route this message–there should be no use in doing so since the connection is about to be closed.

            // Also see special case at top of this function.

            break;

       

        case 2100:    // actual:       New account data requested. API client has been unsubscribed from account data.

            handledBy = accountUpdateReqTracker .error_Route (id, errorCode, errorString);  // make sure AccountUpdateRequest::error_Generic gets this error

            AccountUpdatesCancelledByServer (errorString);

            break;

       

        default:

            handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                // calls ActiveRequestCategoryManager::error_Route which calls ActiveRequest::error_Route for each outstanding request

                // which calls error_SpecificRequest for request with matching id if found

                // and may call error_Generic which is currently only implemented by AccountUpdateRequest

                // for error 2100 "New account data requested. API client has been unsubscribed from account data."

           

            if (handledBy == NULL)

                if (id != kIBNullReqId)

                    (void) Order_error_Route (id, errorCode, errorString);

            break;

    }

   

    return handledBy;

}

So this code errs on the side of NOT routing errors to orders first, except in the case when the error code is specifically known to deal with orders.  The worst consequence of this is not really SO bad because the order callback messages will take care of the basic state of things most of the time, and if an unexpected state change occurs, there is at worst probably a message in the log to explain why, at which point the routing code can be modified as appropriate.

I'm not necessarily advocating this approach, but it is the one I took.

[A] by Richard L King

Regarding partitioning of the ID space, I use the following ranges for

various request types.

Market data requests: 0 – 0x3FFFFF

Market depth requests: 0x400000 – 0x7FFFFF

Historical data requests: 0x800000 – 0x80FFFF

Execution details requests: 0x810000 – 0xFFFFFF

Contract details requests: 0x1000000 – 0xFFFFFFF

Order ids: 0x10000000 – 0x7FFFFFFF

Partitioning the ids in this way makes it easy to identify what type of

request an error message refers to, indeed the error can be routed directly

to the handler for that sort of request.

There is no particular significance in the size or placement of these

ranges, apart from the fact that they are all plenty big enough for all

practical purposes (no doubt someone will prove me wrong by running their

trading client for 514.8 years at 10000 orders per day, but I may not care

by then).

In my API implementation, clients make their requests using ids in a range

from 0 to n, where n depends on the type of request and is derived from the

id ranges . Thus market data request ids are in the range 0 – 4194303,

execution details requests 0 – 8323071. This approach allows clients to use

the request id as an index into an array to locate the relevant handler

object for the response, rather than searching a list or using a hash lookup

(the time saving is really irrelevant, but this sort of thing appeals to my

vanity).

When a request is made, the API code translates the caller's request id into

TWS's id space by adding the appropriate base value, and vice versa when

passing out the response.

Order ids are handled slightly differently in that the client plays no part

in allocating them. The placeOrder method itself allocates the id to the

supplied order object. Note that if the supplied order already has a

non-zero id then this is just an order modification and no id needs to be

allocated. I considered having an explicit modifyOrder method, but decided

against it.

Richard

[Q] Trying to modify an order and sometimes FAIL!?

[A] by Kurt (taken from this thread)

That may be a solution, but you haven't addressed the problem. Here is what

*might* be your problem. Since it is the "standard" symptom that I'm so

familiar with I think it is likely. You would have this problem if you save

an order returned by the openOrder message and use it for subsequent calls

to placeOrder.

It is a bug in processMsg. In C++ the bug is in

EClientSocketBase::processMsg, case OPEN_ORDER.

With every new release of the TWS API I go through it and make the same

fixes. See the following block of 5 lines with my changes marked in two

places:

DECODE_FIELD_MAX/*kkb–was DECODE_FIELD*/( order.volatility); // ver 11

field

DECODE_FIELD( order.volatilityType); // ver 11 field

DECODE_FIELD( order.deltaNeutralOrderType); // ver 11 field (had a hack

for ver 11)

DECODE_FIELD_MAX/*kkb–was DECODE_FIELD*/( order.deltaNeutralAuxPrice);

// ver 12 field

DECODE_FIELD( order.continuousUpdate); // ver 11 field

The problem is that IB's code is defaulting absent order.volatility and

deltaNeutralAuxPrice fields incorrectly when the fields are missing (empty

string) from the incoming OPEN_ORDER message. DECODE_FIELD_MAX rather than

DECODE_FIELD is needed to detect the empty string case and substitute the

appropriate "UNSET" value, in this case UNSET_DOUBLE (which is defined as

DBL_MAX, and thus the name DECODE_FIELD_MAX).

The DECODE_FIELD versus DECODE_FIELD_MAX should probably agree with the

default values set by the Order constructor and in these two cases they do

not, and it causes a problem. Scan the Order constructor for how numeric

fields are initialized. Some are initialized to zero and some are

initialized to UNSET_DOUBLE or UNSET_INTEGER as appropriate for the type.

processMsg should probably be using the UNSET value for missing fields. In

short missing message fields should get the default value that the Order

constructor would have given them. Or I believe that is a reasonable model

anyway.

As a result of this problem, the openOrder message returns a corrupt Order

to you. If you use it with placeOrder it will fail, because placeOrder

wants to see the value UNSET_DOUBLE in fields that are not applicable for

non-volatility orders. Instead openOrder is putting a zero in these fields.

Make the two fixes above and the problem goes away. These are the only two

corrections I make to the EClientSocket stuff. I believe there may be other

places where processMsg disagrees with the Order constructor, and I have

them noted somewhere, but they have not caused me any actual problems, so I

don't bother fixing them on every API release.

-Kurt

[Q] Trying to modify trailing stop, got an error 10067

[A] by Josh View/Reply Online (#39111)

Added on 13-Dec-2017

So it looks like the order didn't immediately trigger and a trail stop price was not assigned at 19:25:02. Therefore it could be modified without specifying a trail stop price. Later, there was a trade in the market which triggered the system to modify the order with a trailing stop price. When an attempt was made to modify the order again (19:25:06), it no longer matched the order in the system because the trailing stop price had changed. To modify the order at that point it would be necessary to use the Order object in the most recent orderStatus() message returned instead of the initial orderStatus which did not show a trailing stop price.

[Q] Order execution messages considerations

[A] by Kurt (from here)

I could be missing something re the "multiple" messages, but will hazard a

guess.

You should never get two execution messages with the same execId. You have

to keep track of such things if you want any reliability. Duplicate

messages should not be a problem whether they are orders or executions as

long as you maintain the necessary information. (Well I can't speak for

what is possible within Excel.)

Some people think it is trouble to keep track of a state for each object

they are concerned with. But the trouble of not doing so is immense. You

create problems that don't really exist. IB is not trying to create the

kind of system that would be reliable to use without keeping track of state

at the other end. So if you don't maintain the necessary information, the

problems you get as a result are essentially problems you created.

You can request executions at any time, and then you will get more

"duplicates", but again just ignore the ones you already received. As I

recall some people request executions explicitly because they are seeing

execution messages getting dropped. (This is a reason to track both orders

and executions since conceivably either might get dropped.)

-Kurt

[Q] Checking API connection / checking the timeliness of ticks

[A1] by rwk2095

> I subscribe to an issue (e.g. ES or QQQQ) that I know should be

> updating.I timestamp all updates including bids and offers,

> and raise an audible alarm if I haven't received an update within

> some time interval such as 10 seconds.

I like the idea, thanks. It leaves out stock specific problems but catching all connectivity and market-specific problems with one little sub is nice.

[A2] by Mike Smith

For entering positions my ATS watches a fast moving average of volume and I consider it a "bad thing" if volume is accelerating or decelerating beyond a certain point (as confirmed by back testing) and I do not make the trade. As a side effect that takes care of cases where ticks stop coming in for whatever reason.

Now for very slow markets you are describing you will have to back test it but you might find the same thing – that a (very) long pause of little or no volume is not the ideal entry point, so the same technique might work. But I would do two thing: (1) Do not use SMART as the exchange, use the highest volume exchange for the market you trade, for stocks that's usually NYSE or highest volume ECN for that symbol. (2) Find a high volume symbol using the same exchange and reqMarketData for that symbol as well but just throw away the data and look for pauses. But then it is as Jim say, what are you going to do? I have a notification system (text msg and email) that kicks in when anything gets too wacky. There are some cases where you just have to be there and make a "human" decision, long pauses in ticks being one of them.

Mike

[Q] AvgPrice rounded to 2 digits, I want 3 digits!-)

[A] read this thread

[Q] How to get avgFillPrice for multiple filled order status updates?

[A] by alex_7880

orderStatus() also returns "int remaining". My logic is basically to wait for remaining to be 0 before I consider the order to be filled, and I use the "double avgFillPrice" that is returned when remaining=0.

[Q] rounding to the tick size of the Financial Instrument

[A] by Mike Smith from this thread

You can automate figuring out the number of decimal places. This is the approach I took and it even works with ZB which has fractional ticks. I have a SymbolInfo class which contains a nearestTick() method that does what you want:

class SymbolInfo {

// This is assigned a value in constructor (below)

private int tickSize;

////////////////////////////////////////////////////////////

// Note: These two class vars are used only by nearestTick()

//

// The value of nearestTickMultiplier is based on the number

// of digits in the fractional part of the tick size. It's

// 10 to the power of that many digits. So if the tick size

// is a penny (0.01) then the number of digits in the

// fractional part is 2 and nearestTickMultiplier is 100.

private long nearestTickMultiplier;

//

// The value of tickSizeMultiplied is the tick size multiplied

// by nearestTickMultiplier. It is used to calculate the price

// as a long int which is the smallest possible long int which

// can be used to calculate the price by converting it to a

// double and dividing by nearestTickMultiplier. This guarantees

// price is a power of 10 without roundoff errors.

private long tickSizeMultiplied;

//

// Another example would be for futures contracts which use 1/32

// as a tick size which is 0.03125. In this case the number of

// digits in the fractional part is 5 so nearestTickMultiplier

// would be 100000 and tickSizeMultiplied would be 3125.

//

// This would also come in handy for representing tick values of

// FOREX symbols where one currency relative to the Dollar might

// be something in the neighborhood of 0.000000000001 to 1 (or

// visa-versa the way things are going lately).

///////////////////////////////////////////////////////////////

// Constructor

public SymbolInfo(ContractDetails contractDetails) {

tickSize = contractDetails.m_minTick;

int n = getFractionDigitCount();

// nearestTickMultiplier is always 1 or greater.

nearestTickMultiplier = Math.round(Math.pow(10.0, n));

tickSizeMultiplied = Math.round(tickSize * nearestTickMultiplier);

}

/**

* Get number of digits in fractional part of tickSize

*

* @return Number of digits in fractional part of tickSize

*/

public final int getFractionDigitCount() {

double fraction = tickSize – ((long)tickSize);

String s = Double.toString(fraction);

int n = s.length();

if (s.contains(".")) {

if (s.equals("0.1")) {

n++; // force trailing zeros, just looks better (GC)

} else {

while (s.charAt(n-1) == '0') {

n–;

}

}

}

if (s.startsWith("0")) {

n -= 1; // remove leading zero

}

n–; // remove decimal point

return n;

}

/**

* Round given price to nearest tick boundary so

* the returned value is a multiple of the tick size

* for this instrument.

*

* @param price The price we want to adjust

* @return The price adjusted to the nearest tick boundary

*/

public final double nearestTick(double price) {

long ticks = Math.round(price/tickSize);

price = (double)(ticks * tickSizeMultiplied);

return price/nearestTickMultiplier;

}

}

[Q] Should I use MKT or LIMIT orders? How do I track partial fills?

[A] by tdrtw from this thread

Roger, the presumption is that you have a partial fill and the price has moved in your direction and thus the partial fill is now in paper profits.

When I manually traded I used to enter at an exit price using a STOP-LIMIT order to specify that price and often had a partial fill in a fast moving market. Sometimes I waited and the price came back and the fill completed.

When I moved to a fully automatic system, I changed my rules and decided to Enter on a STOP order with no Limit Price.

Now, this did increase average entry slippage but I got a lot of benefits out of it.

1) So far every entry order has always filled.

2) Now that the entry orders are filled, exit orders are placed within a split second. Which is much safer than a partial fill open out there and having complex software rules to manage this.

3) I now catch fast moving markets and take profits on orders that once I never got into using Stop Limit entries.

4) Yes once in a while the slippage does go against me.

Overall, I have found no detriment to system performance. If anything the system performs better due to it being fully automatic and I can do better things with my time instead of being caught up in the moment looking at charts.

My system is much safer in automode using a STOP entry than STOP-Limit entry with exact price.

I know it can be hard, but sometimes it's better to modify your system to fit software, than create a complex software program to force it to run your current rules.

Since being automatic I now create much simpler systems that are easy to program and less chance for things to go wrong.

That's just my opinion.

Good luck!

Damien

(aka Exotric_Adriana_1982 πŸ˜‰

[Q] Why PosixMQ is a good idea for IPC? Why can’t I just use some thread-safe queue (with mutex)?

[A] by Nils Pipenbrinck from stackoverflow

In realtime OS environments you often face the problem that you have to guarantee execution of code at a fixed schedule. E.g. you may have a function that gets called exactly each 10 milliseconds. Not earlier, not later.

To guarantee such hard timing constraints you have to write code that must not block the time critical code under any circumstances.

The posix thread synchronization primitives from cannot be used here. You must never lock a mutex or aqurie a semaphore from time critical code because a different process/thread may already have it locked. However, often you are allowed to unblock some other thread from time critical code (e.g. releasing a semaphore is okay).

In such environments message queues are a nice choice to exchange data because they offer a clean way to pass data from one thread to another without ever blocking.

Using queues to just set variables may sound like overkill, but it is very good software design. If you do it that way you have a well-defined interface to your time critical code.

Also it helps to write deterministic code because you'll never run into the problem of race-conditions. If you set variables via message-queues you can be sure that the time critical code sees the messages in the same order as they have been sent. When mixing direct memory access and messages you can't guarantee this.

[A2] by Alok Save from stackoverflow

Message Queues are predominantly used as an IPC Mechanism, whenever there needs to be exchange of data between two different processes. However, sometimes Message Queues are also used for thread context switching. For eg:

You register some callback with a software layer which sits on top of driver. The callback is returned to you in the context of the driver. It is a thread spawned by the driver. Now you cannot hog this thread of driver by doing a lot of processing in it. So one may add the data returned in callback in a message Queue, which has application threads blocked on it for performing the processing on the data.

[some feedback from 100% complete frameworks out there;-]

[A] by poch32

Depending on the security, my ATS currently checks for anything between 3 and 8 stratregies. Strategies are given a "privilege ring" (so to speak in OS jargon) thus, in case of clashing, one will always prevail among the others.

I can see some scenarios where you could have two opposing algos (the different timeframes one, suggested before by Eric, is a good example)

However, in my case, since all my strategies are – more or less – oriented towards the same timeframe, I cannot allow 2+ strategies to collide, since that would be, effectively, giving money away in comms.

If I had two independent, concurrent instances of my ATS running, and I was in the need of allow concurrent strategies to collide, I would clearly go down the pipe/tcpip route. Preferably the last one, since sooner or later perhaps I could need to move one of the ATS to another machine.

[A] by tdrtw

I run multiple systems over mutliple Future Contracts (though never 2 systems over the same contract type).

I created a shell java console app that runs on a Windows XP Pro machine and communicates with an Oracle XE database.

The shell java console app can ONLY trade 1 financial instrument (FI from here on). It is an input paramter when I start the .jar file which maps to a database control table to get all the info such as expiry, how much to risk on the trade, tick size, strategy type etc..

The database control table also tells it what strategy to trade. The java console app makes a generic call to an Oracle Stored Procedure that will then call the correct strategy which is just coded in an Oracle Stored Procedure.

Basically when I boot up each day, I start multiple instances of the java console app each pointing to a different FI which each trades it own strategy.

Took me a long time to get it to work, as I went down the wrong path many times. But one thing I can say, it works f&*ing wonderfully, and as you can tell, I'm a little proud of it πŸ™‚

I had prior skills working on an Oracle database, but never had any skills in Java when I started the automation venture many years ago. So I leaned on my Oracle Database skills and have the code split between java and oracle pl/sql.

Now that I look at it, it's no big deal but it was a little challenging for me at the beginning. Stick to it and it will fall in place. Better to go 10 steps forward and 5 steps back than just stand still.

Also be careful of leaning too much on these boards. I remember once sitting next to an experience person and found myself constantly asking them questions which I could really work out the answers to. Once that person moved, I found myself working things out for myself more and actually suprised myself in a sense that I can actually do it if left to sink or swim.

Good Luck, you will find your path eventually!

Damien

[A] by Richard L King

Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.

2. Develop a completely separate daemon process that runs one or more instances of IBController based on configuration data: eg run IBController on the live account at 06:00 on Monday mornings, stop it (via IBController’s command channel) at 22:00 on Friday, and if the IBController process drops out at any point between these two times then restart it. Note that there is already sample code for doing this on Windows in the IBControllerService sample (though less sophisticated than what I just described).

Actually I already do something very similar to 2. using nothing more than the Windows Task Scheduler. So if for example there’s a power failure during the week, or IBController/TWS crashes (though it never does), IBController is automatically restarted when power is resumed, but not for a weekend failure.

What I don’t do is any kind of active monitoring of TWS or the API (by poking it in some way), because during 11 years of running API programs all day every trading day, I’ve never found any need for it. (Your mileage may vary, though I don’t see why. But maybe if I were pushing hundreds of order per hour through the API, or continually downloading historical data, it would be a different matter.)

I also don’t see any need for trading systems to wait until IBController is ‘ready’ in some sense. My platform doesn’t care in the slightest whether IBController or TWS is running – if so configured, it simply keeps periodically retrying the connect until it succeeds. Indeed I run most API programs on different machines than the one running IBController/TWS/Gateway.


[Q] Unit testing – which one to choose?

[A] by Dmitry (added on 07-May-2015)

Very good article on the C++ Unit Testing

“Exploring the C++ Unit Testing Framework Jungle”

http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle

also see:

http://stackoverflow.com/questions/1407354/is-there-a-c-unit-testing-library-that-is-similar-to-nunit

good wiki on unit testing comparison (it has other languages as well!)

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks


[other chaotic implementation notes]

———————————————————————————————————–

Tick momentum indicator

An Idea by MarketMole (from this thread)

I've also built tick momentum indicators which do indeed try to determine the side that trade fell on as far as the going bid/ask spread was concerned. I know I've run these against various brokers but I'm sure I also ran against IB at some time and (thought) I was able to determine general trade sides, in a broad sense.

———————————————————————————————————–

On topic: “What to monitor [remotely]”

by Jason Sapp from this thread

I do not monitor actual positions through my GUI, but rather just watch my Net P/L, Worst Case Situation, Unrealized P/L, Realized P/L, Position Count (long/short), Missed Positions, etc….  I've thought about putting in functionality to monitor each position, but I just haven't done it.  I also want to eventually implement the capability to close out positions manually (if need be), but (once again), just haven't had to real need to do so.

I've attached a screen shot so that you can see what it looks like (yesterday just happened to be a good day for me, so my real-time graph looks quite nice).  Please excuse some of the anomalies towards the end of my yellow line (on the attached graph), where my Worst-Case exceeds my Net P/L.  It's just the way I'm sometimes capturing data at the moment a position is being closed.  I just haven't taken the time to fix the bug.

3h2dg.jpg

Note: this was drawn using http://www.amcharts.com/demos/

———————————————————————————————————–

It's not intuitive but IB only sends the deltas of price and size, not

complete tick events.

If a trade occurs at the same price as the previous price message, you

will only get a size message (the price is implicitly the previous price

message).

———————————————————————————————————–

rwk:  I only use order status events for logging

zsolt: Final q, it is worth to correlate OrderStatus and Exections data, and make

an alert if they don't agree, in order to confirm that TWS and/or ATS works

correctly, or, Executions data is reliable in practice to trust?

———————————————————————————————————–

I have never missed an execution (WHILE CONNECTED), but others here

have. Although *I* have not missed an execution, others here

have, so IB isn't 100% reliable, just using execDetails.

You should write the reconciliation code, and the replay code,

because you WILL need it if your connection breaks.

———————————————————————————————————–

Nice, thanks. Last week i have created the reconciliation code (even I named

my class as OrderReconciler :), which compares my persisted state in db with

openorders and executions data, and push executions and cancellations to the

alg to update their state in memory and in db.

In this way I can restart the app without losing context, but naturally i

might miss some trading opportunity.

Until now, this seems to work correctly.

Thanks for the suggestions for everybody about execdetails.

———————————————————————————————————–

That's right re updatePortfolio. And your scenario is the same reason I

don't use rwk's approach. However if you are reconciling things against

each other especially *during* trading, updatePortfolio is potentially damn

useful since if any change occurs in updateportfolio that is not predicted

by the order/execution messages you have received, then you know you

probably missed something. Under that condition your code could request

open orders and/or request executions again, and/or produce an alarm.

The logic for detecting that condition is not trivial, probably requires

some time slop, could possibly consider all possible combinations of orders

that *might* have executed or partially executed, etc.

-Kurt

———————————————————————————————————–

[good implementation note on “designing stop-limit orders”]

found here: src

a possible solution:

use both stop order and stop limit order with OCO

for example, you want to buy stop at 10.00

so you put a stop-limit order at 10.00 and a buy stop order at 10.10

let them OCO to ensure only one order will be triggered

———————————————————————————————————–


Well known errors and how to avoid them

[Q] reqSecDefOptParms() with exchange specified returns no data?

I find that reqSecDefOptParms() with a non-empty exchange parameter never returns any data. But if the exchange parameter is blank, it returns data for all exchanges.

 

For example:

 

reqSecDefOptParams(0,"MSFT", "STK", "CBOE", 272093)

 

gives nothing whereas:

 

reqSecDefOptParams(0,"MSFT", "STK", "", 272093)

 

returns data for AMEX, BOX, CBOE2, BATS, ETC.

 

Am I doing something wrong, or is it just broken?

 

Richard

[A] by Josh View/Reply Online (#44803)

Added on 27-Jun-2020

The exchange field is only for futures/futures options, it should be labelled 'FutFopExchange'. For stocks it should be left blank.

Josh (IBKR)

—- also later: summary from Richard —

So to summarise what is still a very illogical situation regarding the futFopExchange parameter of reqSecDefOptParams():

 

  • The documentation says:

"futFopExchange The exchange on which the returned options are trading. Can be set to the empty string "" for all exchanges".

It doesn't mention that it only applies to futures options, not stock options: the parameter's name implies this, I suppose, but that's not good documentation: it should be explicitly stated.

  • If you DO set it to the empty string for futures options, you get "Error 321 … Missing exchange for security type FUT".
  • Because futures options only appear to trade on the same single exchange as the future itself, there is only one possible choice for this parameter for any given underlying. So the parameter is completely useless.
  • If you set it to the empty string for stock options, you get info for all exchanges returned, as expected.
  • If you DON'T set it to the empty string for stock options, you get nothing at all – not even an error!

 

So can anyone see any logic in this? The futFopExchange parameter accomplishes precisely nothing, except cause trouble.

 

I can't help thinking I'm missing something, but I've no idea what…

[Q] I observe gaps in the most recent historical date (options in my case). But later same data looks good. Why?

[A] by Jeff View/Reply Online (#40888)

Jeff, Unfortunately you may be encountering an issue where there is a lag in recording the most recent historical options data to the server database. For tracking status updates it would be best to contact customer service. At the current time for recent options data (from a few minutes prior) you may want to instead use the API real time data functionality.

[Q] Errorcode(110): The price does not conform to the minimum price variation for this contract

[A1] read this thread (it’s only 43 messages;)

[A2] by Nick View/Reply Online (#38922)

The message means that the limit price is not a multiple of the minimum price increment of the stock. For example, if the stock trades in increments of one cent a limit price of 20.153 would not be valid. This is usually due to a price that comes from a calculation, like a moving average.

[Q] Errorcode(162): Error message:No market data permissions (for some tickers, even with proper subscription!)

View/Reply Online (#43032)

I can't download historical data for certain tickers.

Error 162, reqId 9: Historical Market Data Service error message:No market data permissions for AMEX, ARCA, BATS, BEX, BYX, DRCTEDGE, EDGEA, ISLAND, NYSENAT, PSX STK, contract: Stock(symbol='sfet', exchange='SMART', currency='USD')

SFET is foreign company but it is listed on NASDAQ, same happened with IMRN yesteday (also listed on NASDAQ). I don't think i had problem like this before.

I'm subscribed to classic level 1 data. It should work when it's listed on NASDAQ.

[A] by toth.david6@gmail.com View/Reply Online (#43036) 

Added on 04-Oct-2019

Problem solved. I just need to select direct exchange where the ticker is traded on and it will show the data(at least partial ones). These tickers are traded on more exchanges and if i choose SMART and i am not subscribed to every exchange its traded on i will get the error via API or delayed data in TWS platform.

I thought it's little smarter and it will give me data from all the exchanges i am subscribed to and aggregate it accordingly, but if youre missing just one exchange it will give you error right away. That $4.5 level 1 data bundle cover doesn't cover everything in this case.  

[Q] Errorcode(162) Historical Market Data Service error message:HMDS query returned no data: GCQ9@NYMEX Trades

[A] by Josh View/Reply Online (#43045)

Added on 08-Oct-2019

That error most commonly occurs when data is requested outside the date range when the product was trading. When I try it for 7/1/19 for instance it returns data ok.

[Q] Errorcode(162) 'Historical Market Data Service error message:Starting time must occur before ending time'?

I'm downloading 1 second historical bars and got almost a year of data (2000 bars per one request, 10 seconds between requests), but at some point it starts to return mentioned above error, which makes no sense, since in the request there's only 1 "end date" argument, there is no "starting time" whatsoever.

[A] by Dmitry

Added on 08-Mar-2020 πŸ™‚

Consider this as "no more data available". IB can also return same error number 162 with text "HMDS query returned no data", which means the same. If you need more years of history, then consider asking larger bars (instead of 1 secs bar try 1 min, 10 min etc.)

[Q] Errorcode(200) No security definition has been found for the request

by enthalpyyan@gmail.com

Hi Guys,

When I place a option order with the following arguments, I always got 200 error.

        Contract contract;

        Order order;

        contract.localSymbol = "MSFT131206P00034500";

        contract.secType = "OPT";

        contract.currency = "USD";

        contract.exchange = "SMART";

        order.action = "BUY";

        order.totalQuantity = 1000;

        order.orderType = "LMT";

        order.lmtPrice = 20.00;

        m_state = ST_PLACEORDER_ACK;

        m_pClient->placeOrder(m_orderId, contract, order);

How can I solve this problem? Thanks.

[A1] by souqmate@yahoo.com

make sure you use the result of reqContractDetails() on

symbol="MSFT" secType="OPT" exchange="SMART" currency="USD"

In your case, you'd rather use localSymbol="MSFT  131206P00034500".

Specifying symbol="MSFT" is good practice, but doesn't seem necessary in this context (at least for reqHistData).

See also: [Q] futures options data via API

souqMate

ALSO

you don't need primaryExchange = "NASDAQ",

but you need expiry = "20131213"  (YYYYMMDD), as that's the output of reqContractDetails.

play around.

souqMate

[owner reply]

Hi, thanks.. placeOrder() works now with the following:

  contract1.secType = "OPT";

  contract1.multiplier = "100";

  contract1.exchange = "SMART";

  contract1.currency = "USD";

  contract1.includeExpired = 0;

  contract1.localSymbol = "";

  contract1.right = "Call";

  contract1.symbol = "MSFT";

  contract1.expiry = "20131206";

  contract1.strike = 36.5;

        order1.action = "BUY";

        order1.totalQuantity = 1;

        order1.orderType = "LMT";

        order1.lmtPrice = 0.2;

But I get stucked again.. I want to place a combination option order, how can I do this in C++? IB's API doc really sucks and I can't find useful infomations..

——- AND LATER: ——-

IB's API really makes me crazy. With the same settings, I can place an order with MSFT but I can't place with like AAPL or GOOG. It is the same error "no security definition has been found for the request". I am totally confused about how can I place an order.

[A3] by souqmate@yahoo.com

Before you try to place an order, make sure you have output from reqMktData (or reqHistData when markets are closed).  For that, the output of reqContractDetails helps in specifying the contract; eg get 628 contracts for symbol="MSFT" secType="OPT" exchange="SMART" currency="USD".

There might be factor 100 hiding in strikes (UK stocks), etc.

Also, beware that some functions misbehave on the demo account.

Keep us posted once you succeed in your combo.

souqMate.

[Q] Errorcode(201): Order rejected – reason:It does not comply with IB's order handling rules for derivatives that result in physical delivery.

[A] by Kurt

So it may not turn out to be an API question.  At least it seems evident that the problems you describe relate to the actual differences between the accounts themselves.  I'd suggest the next step is probably to do things manually in TWS and then call IB for support as needed, to clarify account configurations and distinctions so you can know what you are working with.

-Kurt

[Q] Errorcode(300): Can't find EId with tickerId:40000001.

Added on 08-Nov-2017

[A] by Josh View/Reply Online (#38925)

John,

The error "Can't find EId" is typically triggered when an attempt is made to cancel a subscription that was already cancelled or with an incorrect tickerID. I'm not in front of a Windows computer but trying the Java API here it works ok. Are you using the edemo account by chance? The real time bars have never worked there, one of the problems of testing with that account.

Josh

[Q] Errorcode(321): Missing exchange for security type FUT

[A] see: [Q] reqSecDefOptParms() with exchange specified returns no data?

Added on 27-Jun-2020

[Q] Errorcode(321): Error validating request:-'rc' : cause – You must specify an account.

[A] by Kurt

The account (or perhaps called m_account) field of the Order must be set if you have an FA login.  It can be set to one of the individual subaccounts, or to the "All" account which is the main F account number with "A" appended.

Also, when I last checked, the account field must be set on the first call to placeOrder for a given order.  It can not be set on a modification.

I think someone said that groups and allocations can be made to work also, but I believe this used to not work in the situation I needed for some reason, so I have not tried it again and am not familiar or up to date on this.

-Kurt

[Q] ErrorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – Historical data queries on this contract requesting any data earlier than 199 year(s) back from now which is 18210405 00:18:14 GMT are rejected.  Your query would have run from 5240807 23:29:46 UTC to 5240807 23:33:06 UTC.'

[A] Well obviously reqHistoricalData is not happy with endDateTime argument value, check it carefully.

[Q] errorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – End date not supported with live updates'

[A] This might happen when you used keepUpToDate=true in your reqHistoricalData() call. I found solution here. Quote: I have mail to ask the IB team. He told me to left endDateTime as blank. It's work!!

[Q] Errorcode(???) Order Canceled – reason:Order size exceeds amount allowed by fat-finger check

by eugene.kononov@gmail.com

View/Reply Online

[A] (29-Mar-2018)  this error can't be overridden through the precautionary settings in TWS/IB Gateway. There is a precautionary setting for order size, but it is separate. If the user contacts IB ahead of time they can request to have the fat finger check modified in their (live) account.

[A] Incorrect answer by rwk was

I'm surprised you haven't had this error before.  On the TWS click:

Edit > Global Configuration > Presets > Futures > Precautionary Settings > Size Limit > [set to zero (default is 5)]

[A2] by Frank Bell

I had problems with this in a paper account today.  I had a short position of 75000 TNA.  I had orders to buy 30000 at market canceled even after changing the presets.  Sometimes it would accept orders at a given size and then later reject.  I think this is just another example of paper account flakiness.

[Q] Errorcode(10225): Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately

[A] found here https://groups.io/g/twsapi/topic/69943076 and https://170.106.50.235/mementum

Up to the TWS API v972, IB realtimeBar subscriptions could stop working during the trading session and no error notification was sent to the application. This happens because of what's called 'busts' events.

Quoting the response in "TWS API Users Group" question:

"The busts occur when the datastream is interrupted or modified in the middle of a bar and the subscription is cancelled by the backend. They should be relatively uncommon but unfortunately no can't be avoided completely"

In TWS API v978,a special error code was added to notify the application about the bust event so that the subscription could be renewed. Below is the example of such error message (from TWS Gateway API logs):

15:24:00:896 -> 4-2-16777221-10225-Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately.-

Update: after a week of testing – realtimeBars seems to be working flawlessly. Re-subscribing after bust events doesn't require any delay to be added by the application – just make sure to cancel the previous subscription before re-subscribing, as was mentioned above.

[Q] Errorcode(1101, 1102): Error validating request:-'rc' : cause – You must specify an account.

[A] by Ed Gonen ed.gonen@gmail.com   (added on 30-Jul-2015)

From my experience the midnight TWS disconnects happen at the first 30 minutes after the midnight. I think very close to the 20th minute. Your 5:21 London exactly matches that situation. My logs usually show the 14th minute – i.e. 12:14.

I don't think you need to disconnect from TWS in this situation. At least – I don't. The code indicates a disconnect between TWS and IB servers and TWS always automatically takes care on that. This lasts about 1 min. and you get 1102 code. This is based on many years experience with IB.

——————— also see  API Message Codes  —————————-

System Message Codes

Code

Description

1100

Connectivity between IB and TWS has been lost.

1101

Connectivity between IB and TWS has been restored- data lost.*

Reminder: when you get 1101, you have to re-establish all your market data connections

1102

Connectivity between IB and TWS has been restored- data maintained.

1300

TWS socket port has been reset and this connection is being dropped.

Please reconnect on the new port – <port_num>

*Market and account data subscription requests must be resubmitted i.e. when you get 1101, you have to re-establish all your market data connections


[Q] Error: 2109 Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed.

[A] by Josh View/Reply Online (#40645)

It is actually just a warning to let you know that for the designated order type and exchange there is no distinction between rth and 'outside rth'. It will not prevent an order from being placed. For stocks, this is because regular stop orders don't trigger outside of the regular trading hours period 9:30-4:00. This is to protect traders, as the extended trading hours can be very illiquid and the usual NBBO rules don't apply. You can however place a stop limit order which would be active outside of rth (though there's a reasonable chance of a very bad fill with this combination).

 [Q] error 2137 The closing order quantity is greater than your current position

[A] by Scott Hodson shodson at gmail com

Added on 02-Aug-2018

View/Reply Online (#40990) 

For future reference message codes documented here:

https://interactivebrokers.github.io/tws-api/message_codes.html

2173 = Cross Side Warning: This warning message occurs in TWS version 955 and higher. It occurs when an order will change the position in an account from long to short or from short to long. To bypass the warning, a new feature has been added to IB Gateway 956 (or higher) and TWS 957 (or higher) so that once can go to Global Configuration > Messages and disable the "Cross Side Warning".

[Q] Order size is too large IB limit is 35000

boy, I hope I’ll face it some day… :))

[A] by Frank Bell (from this thread)

Size limits vary based on exchange, legal, and IB internal limits. If you are

bumping up against an IB limit they might change it for you. Contact IB.

[Q] First order after IB GW API connection causes disconnect and order is ignored

Added on 27-Apr-2015

[A] this thread

[Q] Historical Market Data Service error message:No historical market data for XAUUSD/CMDTY@CMDTYBBO Last 60

[A] View/Reply Online (#44172)

Added on 03-Apr-2020

Commodities don't have trades data ('Last' in the error message) so you have to choose bid, ask, or midpoint as the data type.

If you check in the TWS chart you'll see that a different data type is selected and 'Trades' is not one of the options.

[Q] While trying to modify submitted order getting: "Unable to modify this order as it is still being processed."

Added on 18-Jun-2020

Hello,

I've run into an issue when trying to modify an order's limit price in quick succession after initial entry.  The error is:

"Unable to modify this order as it is still being processed."

While I do keep track of the latest limit price submitted on my end, I have no way of directly ascertaining what the last accepted value was in TWS.  (Other than receiving the error message and deleting my last recorded limit price.  This seems like a terribly convoluted approach and ripe for errors.)

So, there are two ways I can think of addressing the issue below (in order of preference):

1. Get the last known m_lmtPrice from the API.  openOrders() provides it but that doesn't fire after each newly submitted order.  orderStatus() does fire after each new order but doesn't appear to include a way to get m_lmtPrice.  Is there another way to get it?

2. I can forget tracking the limit price directly and simply wait until it's "safe" to send a modification based on the order status.  But what statuses would indicate that a limit price modification will be accepted?  PreSubmitted? Submitted?  Inactive?  What if I want to modify the limit price a second time…will the same logic work?

Appreciate any insight on the above.  Part of me is hoping that I'm being a knucklehead and missing a simple solution staring me right in the face.

[A] View/Reply Online (#44735)

In case anyone runs into the same issue in the future, I'm going to post what I found here for posterity's sake.  Essentially, I couldn't find a way to reliably get the order limit prices without constantly calling reqOpenOrders().  That was too inelegant for the way I'm set up, so I went with #2 above and waited for the order status to go "Submitted" before attempting modifications.  That alone was sufficient for my purposes, so I didn't tinker with "PreSubmitted" or "Inactive".

—— Also ———-

In case it helps, if you are using stop limit orders it is normal for the status to stay at PreSubmitted until the order is triggered (you will see a 'whyHeld' attribute with value 'trigger'). In this state it is possible to safely modify the limit price, just FYI.


ContractDetails

[Q] Exchange list in ContractDetails?

I'm trying get exchange list for future symbols using this code:

Contract contract;

contract.symbol = "ES";

contract.secType = "FUT";

client->reqContractDetails(reqId++, contract);

Then in the callback contractDetails(), when I printed contractDetails.validExchanges, it shows: "GLOBEX,MIBSX".

Whtat is this "MIBSX"?  Also why is "CME" not listed (if you try to add ES in TWS, it tells you both GLOBEX and CME)?

thanks,

–Zhao

[A] by rholowczak (from this thread)

GLOBEX is CME's electronic market: http://www.cmegroup.com/globex/


Placing Orders (common)

[Q] I sent an order, what should I expect?

[A] by skunktrader2001

After you sent your order TWS will ASYNCHRONOUSLY send you one or more orderStatus() and openOrder() callback messages as the order is processed (PendingSubmit -> PreSubmit -> Submitted -> Filled), as well as execDetails() callbacks as the orders fill. You will need to capture these state messages from the callbacks for each order in some type of data structure, which you can then check on each iteration through the loop to see if there have been any fills / cancels etc.

————— Also by Richard L. King ———–

Regardless of what TWS logs, your own app should log every event concerning orders: thus, log placeOrder, cancelOrder, and the openOrder(), orderStatus(), and execDetails() callbacks. And you absolutely must log errMsg() events. You should be able to determine from your log exactly what the current state of an order is.

[Q] How do I query the status of my orders?

[A] by Dmitry

Basically you don't. IB's API docs: orderStatus()

This f-n is called by API for you (not by you:) whenever the status of order changes and/or fired after you reconnect to TWS.

Also see the following example demonstrates how the orderStatus() event behaves when there is a partial fill of an order.

Also see execDetails(), which you can request by reqExecutions()

——-

Also: Answer by Ewald (View/Reply Online (#189) from https://groups.io/g/insync/

What you are looking for is the openOrders (or openTrades which has more information) and postitions methods. These methods are much faster and also more reliable since reqOpenOrders can report stale information. In general it's best to not use reqOpenOrders, reqPositions or reqExecutions; instead use the non-request variant.

[Q] How to distinguish Client ids from API vs non-API?

[A] by Josh View/Reply Online (#41871)

Added on 04-Mar-2019

Orders placed by other routes such as IBKR mobile or manually from TWS will always have client ID = 0. Unlike orders placed from the API, they will also have an API orderID of 0 unless they are bound from the API. If you want to distinguish your API orders you could set the client ID to a different value when you connect (unless your API client needs to modify/cancel non-API orders) or use the orderRef field in the Order object to label them with a String.

[Q] Where to Set GTC in placeOrder API?

[A] by rwk2095 (src)

It's part of the IOrder object:

timeInForce() As String

The time in force. Valid values are: DAY, GTC, IOC, GTD.

http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm

[Q] How often can I modify an order?

[A] by Rob Terpilowski rterp99 at yahoo.com

Added on 18-Feb-2016

IB will freeze your account if you send too many order modifications relative to the number of actual executions you are getting.  I had an MM strategy that was doing about 15+ order modifications for every 1 execution it received.  IB sent me a notice within a few hours of it running saying that if I didn't disable the strategy they would be disabling all orders originating from my account.  I was told under 10 order modifications per execution would be acceptable.

[Q] This account cannot have open orders(including Combo) on both sides of the same US Option contract in related accounts.

[A] by Josh View/Reply Online (#43818)

Added on 15-Feb-2020

It sounds like you're interested in having orders held on the IB server as 'inactive' and only validated by the server for all possible errors at a later time, not at the time of submission. Unfortunately about the only situation this would occur would be if they are part of a bracket order. In other cases orders will be checked immediately and rejected if there is a problem such as existing orders on the opposite side of the same option contract, even if there is some condition attached to the order preventing it from being submitted immediately to the exchange.

If the orders are complicated and can't be part of a bracket order the only possibility is to have them held on your local machine, as you describe. If you're concerned about the stability of your connection you could possibly use a dedicated virtual server, such as AWS. Or you could use similar options on the same underlying for hedging if that could work with your strategy.

[Q] How to change the limit price of an order without changing order size after partial fills?

… to change the price of the order, I would place an order with the same order ID, with a different price, but keeping everything else the same.  But in case of partial fills, what happens if the quantity is the same with the changed order?…

…..Basically – do I need to adjust the quantity of subsequent changes to the order depending on how much quantity has already been filled?  Or is the quantity always the total of this entire order?…

[A] by Josh jb201448 found here

Added on 26-Apr-2016

Hi Yu,

Yes the quantity of an order is the total quantity, including those contracts which have already filled.

So if 50 of 100 shares fill and you want to just change the limit price of the remaining 50, but not the number of shares remaining, you would just adjust the lmtPrice and the totalQuantity value would remain 100.

-Josh

[Q] I’m getting multiple filled orderstatus updates

[A1] by vanx23@outlook.com

The "Filled" order status may be triggered multiple times for the same order.

One example of that is when IB's routing logic decide to split your original order into smaller amount that would executes in a short burst. The end result is you will have your order filled as a whole, but will have an "Filled" order status for each sub-parts that are been filled, with the filled amount showing as the original order size.

If the filled size match the original order size, then yes, you can use a flag to avoid processing the redundant "Filled" order state.

[A1] by Ed news1000@edshome.net

Also note that there can be duplicate order status events – you have to detect these yourself.

It might be easier to use the Execution callback instead. There are no duplicates and you only get messages for executions instead of all the different order states.

[Q] Ok! Order executed! But what was the fill price?

[A] by Vlad Palnik (it was offtopic on this thread)

see execDetails().execution.m_avgPrice

[Q] Order Id mismatch between API and TWS

Can someone please explain why the Order Id displayed in TWS order's attribute is different then the order Id which is returned from PlaceOrder command in API? How can it be matched?

[A] by Josh View/Reply Online (#42922)

Added on 14-Sep-2019


Unfortunately the API Order ID is not available within the TWS display, only in the audit trail. The 'permId' is the order ID which is assigned by TWS after an order is placed, and will be unique across the account.

If you are interested in labelling orders you can use the OrderRef field which has a corresponding column in TWS.

[Q] Why my order has “Inactive Order” Status?

[A] by Kurt (src thread)

First, if you requested to *transmit* an order but it comes back as

Inactive, you were probably ignoring the results from the error callback.

Conceivably this could happen without an associated error (you never know

with IB) but I haven't seen it.

IB's treatment of order states and reporting is only barely documented, has

changed over time, arbitrarily without notice and with no change to

documentation.

That said, Inactive may mean more than one thing.

(1) Inactive used to always mean basically one thing: the order is not

active yet. This used to include orders "placed" with transmit false, for

which orderStatus messages used to result. Such orders no longer produce

orderStatus messages. When IB changed that it broke my code. As a result I

had to create my own order state to represent orders placed with transmit

false. It is not ideal because I like the acknowledgement that the order

was received (by TWS) with certain fields pre-checked and more likely ready

to go. The current behavior weakens some aspects of reliability.

(2) Inactive also used to be used for orders that were transmitted but

rejected (e.g. due to some bad field) but reusable, i.e. you could correct

and retry the transmit. Useful for interactive apps or automated apps that

provide interactive fallback. I believe some order rejections still work

this way but I'm not 100% sure.

(3) Inactive can now mean something else: the order was rejected and the

system (for no good reason) is not letting you correct and re-transmit it.

The order id is dead and can not be reused. IB introduced this new behavior

some time in the last year.

In at least some cases you can distinguish (2) from (3) based on the error

code.

My code currently uses an artificial order status string "Rejected" for case

(3), and sets that order status if error code 201 comes in. Error 201

produces the message:

"Order rejected – reason: …"

Status "Rejected" in my interactive app basically tells the user not to

bother correcting the order and re-transmitting. (I should permit correct

and automatically generate a new order id.)

You might prefer for simplicity to just waste the orderId and create another

one. That is probably more likely to continue working if IB continues to

mess with things.

In any case if an order is Inactive you do not need to cancel it.

To understand more in a particular scenario you can just try things that

produce errors, and look at if/how the order appears in TWS as well as any

code from the error callback.

-Kurt

[Q] How to cancel all orders?

[A] by rholowczak (form this thread) (added on 16-Jul-2015)

The "nuclear option" is reqGlobalCancel()

[Q] How to cancel all open orders and close all positions?

[A] by Josh View/Reply Online (#40698)  (added on 12-Sep-2015)

Please note there is actually not a single function to 'close all positions' from the API. It would be necessary to receive the current list of positions from the position() or updatePortfolio() callbacks, and then create and place closing orders. There is a reqGlobalCancel function to cancel all orders.

[Q] How to pause order (like in TWS)?

[A] by Josh View/Reply Online (#39988)

Added on 06-Jun-2018

Unfortunately there's no way an order can be paused from the API. It needs to be cancelled and re-placed. This is essentially what TWS does behind the scenes anyway.

Josh

[Q] How to get order history?

[A] by Josh View/Reply Online (#43808)

The TWS (socket) API provides trading history for up to a week, IB gateway is for the current day only; if you need history further back than that you could use flex queries (which don't require username/password) or the CP API.

[Q] How to track ALL orders (including cancelled orders, and including manually created via TWS when my program was not running)?

Hello,

I am trading both manually and via the TWS API.

In order to analyse my trades better, I am looking for a way to

list orders that I cancelled either manually or via the API.

My problem is that the API-connected program is not running

all the time, so when I do the occasional manual cancel of a STP-order

it is not recorded by the API-connected program.

I could not find a way to export cancelled order Information from TWS.

Ideally I would like to export a list of orders like the one via Account – Trade Log

Menu entry in TWS.

Does anyone have an idea of how I could get a consolidated list of

all cancelled orders within a timeframe, meaning both manually cancelled

and automatically cancelled orders?

As I said, just protcolling them via the API-connected program does not

work, since it is only started sporadically throughout the day. Sometimes not

at all (e.g. days, when I only trade manually).

Thank you very much for any help in advance.

Best,

Josef

[A] by Josh View/Reply Online (#39971)

Added on 06-Jun-2018

Josef,

To receive order history I would also suggest taking a look at the audit trail, it should be easier to parse than the log file and will show order cancellations.

https://www.interactivebrokers.com/en/software/tws/usersguidebook/realtimeactivitymonitoring/audit_trails.htm

-Josh

——- update from Josef: ————

I know this is getting off-topic regarding the API, but I thought this

might be helpful for someone with a similar problem:

The audit data, as Josh has generously indicated, can also be

found in separate xml-files named

Mon.audit.xml etc. for each weekday.

They can be parsed easily and the benefit is that this can be done

by a script by just accessing the files. So one does not have to open

the audit menu in TWS which can be less automized (though I am sure

the marvelous creators of IBController could even automate this).

All the audit menu item does is create an html file from the data in the

xml-files which is then auto-displayed in the standard browser.

The files can be found in the infamous folders under the jts-folder

that specify accounts by a series of letters.

They are overwritten on a rolling basis, so saving them via a cron-job

makes sense.

Thanks again,

Josef

[Q] I got multiple execution messages.

>>> I tried this morning and still got multiple execution messages for

>>> single executions – sometimes. Sometimes it was one execution message

>>> for one execution. But I did get fewer execution messages than order

>>> statuses (stati?)

>>>

>>> Anyone else get multiple execution messages? If they get doubled, they

>>> seem to be 3-5 ms apart.

[A] by Kurt (form this thread)

>> I could be missing something re the "multiple" messages, but will hazard a

>> guess.

>>

>> You should never get two execution messages with the same execId. You have

>> to keep track of such things if you want any reliability. Duplicate

>> messages should not be a problem whether they are orders or executions as

>> long as you maintain the necessary information. (Well I can't speak for

>> what is possible within Excel.)

>>

>> Some people think it is trouble to keep track of a state for each object

>> they are concerned with. But the trouble of not doing so is immense. You

>> create problems that don't really exist. IB is not trying to create the

>> kind of system that would be reliable to use without keeping track of

>> state

>> at the other end. So if you don't maintain the necessary information, the

>> problems you get as a result are essentially problems you created.

>>

>> You can request executions at any time, and then you will get more

>> "duplicates", but again just ignore the ones you already received. As I

>> recall some people request executions explicitly because they are seeing

>> execution messages getting dropped. (This is a reason to track both orders

>> and executions since conceivably either might get dropped.)

>>

>> -Kurt

> I wasn't logging execID; I'll start looking at that. There doesn't

> seem to be a corresponding unique identifier for order status – is there

> one?

>

> I do monitor all state on my end and keep my own copy of the info. I

> imagine it would be crazy-making to try it any other way. I don't

> consider a trade closed until I get both the order status update of

> 'filled' *and* the portfolio update that shows the correct position.

>

> I have never sent a call to get executions – they just show up after

> each order.

The order id fields tell you what you need to uniquely identify the order

you placed and associate the resulting activity with that order. Both

orderStatus and openOrder messages are involved, and there is a little

redundancy between the two, as I recall. (As I recall, they always come in

pairs, and always in a certain order.)

Sounds like you have a generally good approach. I was going to mention

portfolio updates, but didn't want to get into it. πŸ˜‰

You may have to request executions or open orders if you have to quit and

restart your app, depending on what state you save to disk. Or you may want

to do it if you find you are missing executions or orderStatus messages, as

many people here have reported happening.

-Kurt

[Q] How to submit a sell order as soon as a buy order is executed

Hi all,

I would like to submit a SELL order as soon as my BUY order has been executed…is there a way to submit this logic in one shot or do I have to listen to the execution of the BUY order in order to submit the SELL order?

any help would be really appreciated…

Stef

[A] by tdrtw from this thread

I haven't looked into this myself yet, but I believe there is bracket orders which may suit what you want to do.

I presume you know what price your entry will be and know your initial stop point BEFORE it happens. You can tie a bracket order which will be your stop to your entry order. The bracket order will not come into effect until your entry order is fired.

Let us know what you find out.

Cheers,

Damien

[A2] by Ed

You want parent/child orders. The parent is your entry and the child

is the exit. In the child order you set ParentId to the entry order.

There may be some discussions in the archive. I have used this and it

actually works (not always the case with the api).

[Q] How do I know if my stop order is triggered from API message?

[A] by Richard L King (found in this thread)

You get orderStatus and openOrder callbacks when a stop limit order is triggered.

You should read the definitions of the various statuses in the API documentation to find out which status applies (I think it changes from presubmitted to submitted but you’ll need to verify that).

When it comes to orders, you should be logging absolutely everything that happens – you can’t afford to miss anything where your money is at stake. If you were doing this, you wouldn’t have needed to ask the question.

Regarding how to get the best execution price with a stop limit order, I doubt anyone knows any better than you, and it probably depends on whether you’re trying to get into a position or out of one.

Richard

————————————— also ——————–

Joe: Richard,

You are correct. The orderStatus change the 'PreSubmitted' to 'Submitted' once the stop order is triggered. This something I missed. Thank you for the information.

For the limit order execution issue, I set up an stop limit order for cut loss instead of using stop order which usually results in a very poor price. So, do you think it is a practical approach to create a thread which keep comparing the last price with my limit order price if the last price cross my limit price, we re-submit the limit order with new limit price which could be possible executed. I am pretty new to this, your help is best appreciated.

Best,

Joe

————————————– and finally ———————-

Your suggestion is one possible way, though in a fast-moving market you

might find that you are chasing price movement and never quite catching it,

and you end up incurring a much bigger loss than if you had used a stop

order. Bear in mind that the prices you're getting through the API may

easily be 100+ milliseconds behind the market, and it could easily be

another 100+ milliseconds before your amended order reaches the exchange.

Another possibility is to set the limit price of the stop-limit to be the

worst price you're prepared to accept, and the chances are you'll get a

better fill than that if the market is currently higher (for a sell) than

your limit (in which case a market order would have done just as well). But

again, you may not get filled at all.

In a stop-loss situation the important thing is to be out of the position.

If the exchange offers a native stop order, my view is that it's the best

option – it will get the best price that's available at the time. There may

be a better price available a microsecond later, but there's no way you

predict that.

Where the exchange doesn't offer a native stop order, IB usually has a

simulated stop order which automatically does rapid order amendments with

stop-limit orders in the way you described – but much more effectively than

you can do it yourself, because IB are very close to the exchange.

If you're trying to get out of a profitable position, rather than cutting

your losses, there is perhaps more room for creativity. You could have a

limit or limit-if-touched order to extract the best profit, and a stop or

stop-limit order as a backstop: you could then adjust both of them

periodically, perhaps converging them together – but not too close, or you

might find they both execute (even if they are in an OCA group).

The possibilities are endless, and I pretty much guarantee that there are

others on this forum who will have a different view from mine. There's no

'right' answer.

[Q] Should I use IB’s trailing stop and other advanced order types?

Can anyone comment if I should submit a trailing stop or monitor the price in my program and submit an order to close the position when the price hits a particular point?

[A1] by Jason Sapp

My personal preference is to always have a stop order in the market (or at least simulated at IB's back-office), versus just exiting when price hits a certain point. If my ATS goes down, then I know I'm still protected. Lastly, some of the trigger methods that IB supports take a little time to program correctly and they've already done it for you, so that's worth something as well.

Now, with all of that said, I have implemented my own trail, but I still use regular IB stop orders. It's just that I change the price when my algorithms dictate.

Jason

[A2] by Ed <prize@…>

I'll second that idea.

I use the parent/child facility to specify my entry order, profit exit

and stop exit all at the same time. Either I will have no position or I

will have a position with a profit and stop. Should my ATS be 'down'

(anything from program crash to power failure or isp failure) I will not

be exposed to any disastrous drawdown.

As has been mentioned, you don't have to post your orders at the price

you ultimately intend. You can adjust things as you go along (although

not every aspect of a placed order can be changed).

For me it winds up being the both of best worlds, at the cost of a

little more programming. As always, only you can decide which approach

is best for your particular needs.

[Q] How to let an order to execute at a particular time ?

[A]

jimshaw4585: One can combine "good after time" and "good till time" to create a narrow window.

tdrtw: I use a "Sell on Market, Good After Time set to 1 minute before market close" to liquidate any open position grouped in a One-Cancels-All with my other exit orders. In the last 4 years it has worked as anticipated.

[Q] OCA orders?

[A] by Kurt (from this hread)

Orders don't cancel each other unless you put them in an OCA group. That's

what an OCA ("One Cancels All") group is for.

I don't see you setting the ocaGroup field anywhere.

Also don't forget to set ocaType according to what you want.

-Kurt

——————— but then also this: —————————-

When you create a bracket order (ie an entry order together with a related

stop order and/or a target order), you don't have to set the OCA group,

because TWS does it for you (ie orders that have parentId set are

automatically placed into an OCA group). Indeed if you do set the OCA group

TWS overwrites it with its own value.

Bruce, it would be a lot clearer (though not strictly necessary) if you

created a new order object for each order. For the stop and target orders

you need to set parentId to the orderId for the entry order.

Richard L King

todo: give couple working examples of OCA done manually (by Kurt) and by Richard

using a parent/child arraignment

automatically creates an oca group.

————————————— and then very important from Richard King ———————-

We are talking about two different things here.

What I mean by a bracket order is an entry order with linked stop loss

and/or target orders. TWS and the IB servers treat these specially: the

entry order is placed with the exchange, but the stop loss and target orders

are held back until the entry order fills. When it does, they are submitted

to the exchange.

Moreover, if you set transmit=false for all except the last order in the

bracket, then TWS doesn't submit the entry order to the exchange until the

whole set of orders is ready.

The advantages of this are:

1. If your entry order is not filled, then the stop loss and target orders

never get placed with the exchange, so there is no worry about them

executing.

2. It avoids the situation where the entry order executes immediately (eg a

market order or a marketable limit order) and then your connection or your

application breaks before you can submit the stop loss order, leaving you

with an unprotected position.

3. If you cancel the entry order, the stop loss and target orders are also

cancelled automatically.

What you appear to be doing is merely creating an OCA group (for which of

course you have to supply the OCA group name). But this gives you neither of

the benefits of the 'special' bracket order. So if you use a limit or

stop(limit) order to enter, it may never be filled, but there's nothing to

prevent one of your other orders executing (which one would of course depend

on your entry order type).

To create this special order group, you simply have to set the parentId of

the child orders to the orderId of the parent orders: TWS does all the rest

for you.

Note that you can also add the stop loss/target orders after the entry order

has been placed (but before it has executed) using parentId.

By the way, just for completeness, I haven't actually checked recently that

TWS still overwrites any OCA group name you supply when creating a bracket

order in this special way, but it certainly used to. The point is that you

don't need to supply it.

Richard L King

[Q] OCA orders are simulated orders?

[A] by Josh View/Reply Online (#43747)

Added on 07-Feb-2020

OCA groups are a simulated order type not supported natively by exchanges. The orders in an OCA group will be held on the IB server until it thinks one is likely to fill and then it sends that order to the exchange. So you're really only submitting the orders with an OCA group tag to the IB server, not to the exchange. In the case where an earlier order in a group placed milliseconds apart has already filled you would expect later orders in the same group to be rejected, so that is the expected behavior. You can use the Order.ocaType field to choose whether partial fills of an order should cancel or reduce other orders in the group.

[Q] Code: 201 – Order rejected – reason:The OCA group order has already been filled

[A] by Richard L Kins, Josh and others (from this thread)

<tread too long to be copied here.. See online…>

[Q] Order overfilled?

[A] by rwk (from this thread)

I have gotten overfills on USA stocks, though not often.  In my case, it usually happens when I do a cancel & replace.  Sometimes the execution reports are late, and that has been a serious problems lately as I mentioned in an earlier post.

The problem you're describing sounds like a bug in paper trading, assuming you placed only one order and never modified it.

[rwk]

[Q] SHORT selling (SSHORT vs SELL)?

Added on 27-Apr-2015

Hello,

I have been testing my day trading algorithm on the paper account, and I am able to short correctly by simply issuing a "SELL" order. However, in the API manual, it says the order must be one of "BUY" "SELL" or "SSHORT". I was assuming that the "SSHORT" designation was for short selling. However, when I issue that order type in my paper account, I get an error from the API. As far as the paper account goes, using "SELL" works fine for short selling. My question is, will using "SELL" orders for short sales work correctly on the live account? Or do I somehow have to go back to "SSHORT" orders for short sales on my live account? Thanks.

Roy

[A] by Jason Sapp (from this thread)

SELL orders work just fine for US Stocks.  I've been doing it now for approximately 4 years.

Jason

[Q] Changing Order Quantities while order partly filled.

Hi Guys,

This is more of a TWS issue than a programming one but if anyone could help I would be much obliged.

If (say) a limit order is entered for 10 futures contracts and after 6 contracts have been filled I decide that that's in fact all I want how do I change the order to reflect that without actually cancelling the order? From my experience if, after filling the 6, I then amend the Quantity field to 6 it continues to fill up to 10; the same with any amount that I amend the order to lower than the 6 already filled. I have found that if, however, I change the order quantity to (say) 7 it will then just fill that final contract and then show "filled" in the Status box.

The reason I do not want to just cancel the order is that the order is part of a basket that has child orders attached to it and if the initial order is actually cancelled rather than amended then the associated child orders would cancel too – which I do not want.

If anyone could help it would mean a few less grey hairs this end.

Many thanks, John

[A]  by Jsa Jsa

I've figured out a way of doing it which keeps the parent and child orders and works pretty well too; just in case someone has the same problem in the future.

Solution: If the original parent order is to buy 10 contracts and while the order is filling it is decided that (say) 6 is enough, then move the original parent order to a few ticks below the associated child Stop Loss order and change the parent order to just 7 contracts. At the same time also change the amounts of the child orders to (in this case) 6. Then when the child stop or take profit order is executed then cancel the remaining part of the parent order that is left of the original.

I have set up my order entry system to do this automatically but that's what it instructs TWS to do and as I say it works well.

[Q] How do I get commissions from API?

[A] by Brant Hahn

take a look at the openOrder callback on EWrapper – the OrderState object that gets passed in has a property called m_commission on it that will give you a commission value a second or 2 (or more) after your order gets filled.

(update 29-Mar-2018)

This is true, but an orderStatus of filled is not guaranteed. A reliable way to receive commission information is to monitor the commissionReport function, it will have commission information both immediately after the trade and later, in response to reqExecutions. To receive commission information from all API clients it will be necessary to set the API client as the master client.

[Q] Also if I want to see the sum of Commissions in a given day from API ?

[A] by Brant Hahn

you'll have to collect that info yourself from the single trade commissions

[Q] How to check the date when the current position was acquired?

[A] by Kurt

Nothing via the TWS API. You'd have to download statements.

[Q] Can I use just conId for specifying the contract?

[A]

Look at

http://www.interactivebrokers.com/en/p.php?f=programInterface&ib_entity=llc

i.e., the release notes for version 9.64:

"You can now request market data and place orders from the API using the conid, the unique contract identifier. Previously, the conid could only be used for contract details. "

Does this answer your question?

[Q] My order’s place in line!?

[A] by Frank Bell

I've got no special knowledge, but I don't think IB's routing discriminates

between customers. I've had unable orders that should have filled based on the

time they were placed. Naturally I'm unhappy when this happens.

After some email with the SMART group I learned that if, for example, you have

an order to buy 5000 xyz at 5 that's resting on Nasdaq and it becomes possible

to execute 100 on ARCA, the full 5000 will be routed to ARCA in the hope that

the 100 represents the tip of an iceberg order. Of course, once pulled from

Nasdaq, the order has lost its time priority if it's routed back or your price

could be traded through before the order can be routed back. I don't know if

this is the best strategy. I'm believe it's not for very large orders. A

better strategy would seem to be to split the order, routing part to ARCA and

leaving the rest on Nasdaq. But I think that's too much to expect from SMART.

Since there's no guarantee SMART will leave an order undisturbed once initially

routed, there's no way to know where your place in line is.

[Q] Max usage of buying power for DT system

[A]

  • Hi
  • Before running out of account buying power for a DT system, what else
  • conditions should be checked except for buying power itself bigger than
  • zero? I got the following order rejected error msg before running out of
  • buying power. Any suggestions are really appreciated, Thanks,
  • Id: 919 | Error Code: 201 | Error Msg: Order rejected – reason:YOUR ORDER IS
  • NOT ACCEPTED.
  • IN ORDER TO OBTAIN THE DESIRED POSITION YOUR
  • EQUITY WITH LOAN VALUE [95269.89 USD]
  • MUST EXCEED
  • THE INITIAL MARGIN [95902.93 USD]
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Aug 16, 2010
  • View Source
  • I don't think about buying power very much, since I think in terms of
  • margin. But I believe buying power reflects the amount of an ordinary
  • marginable security (e.g. stock) you can buy on margin. So I don't think it
  • applies very well in any more complex margin situation, e.g. involving
  • option strategies.
  • If you are buying stock, then I'd think the simple buying power concept
  • probably applies, and in that case I don't know what would create an
  • exception. But you need to understand margin *completely* regardless. So
  • I'd suggest going into Report Management and producing a margin report and
  • making sure you understand it fully. Unfortunately that is an end-of-day
  • report and may not help you with day trading. In that case you may need to
  • look into the functionality with TWS that may help you analyze your margin
  • situation dynamically. I'm not terribly familiar with those features at
  • this point however.
  • -Kurt
  • Show message history
  • Reply
  • Jason Sapp
  • Aug 16, 2010
  • View Source
  • Hey weidong,
  • I use a fairly complicated position sizing alg, but it starts with an assumption of not risking more than .25% of account equity (including commish and estimated slippage). As i start to exhaust my buying power as the day wears on, i start to reduce subsequent position sizes by a certain percentage. As i exhaust more and more buying power, i continue reducing the size of new positions, until i hit .1%. At that point, i stop reducing buying power for subsequent positions and then just wait until i exhaust buying power. Its taken me a few years to optimize this algorithm, but it works pretty well and i only run out of buying power about once per month.
  • Jason
  • —–Original Message—–
  • From: Weidong Ruan <wdruan@…>
  • Sent: Monday, August 16, 2010 5:18 PM
  • To: TWSAPI@yahoogroups.com
  • Subject: [TWS API] Max usage of buying power for DT system
  • Hi
  • Before running out of account buying power for a DT system, what else
  • conditions should be checked except for buying power itself bigger than
  • zero? I got the following order rejected error msg before running out of
  • buying power. Any suggestions are really appreciated, Thanks,
  • Id: 919 | Error Code: 201 | Error Msg: Order rejected – reason:YOUR ORDER IS
  • NOT ACCEPTED.
  • IN ORDER TO OBTAIN THE DESIRED POSITION YOUR
  • EQUITY WITH LOAN VALUE [95269.89 USD]
  • MUST EXCEED
  • THE INITIAL MARGIN [95902.93 USD]
  • [Non-text portions of this message have been removed]
  • [Non-text portions of this message have been removed]
  • Reply
  • elenosandoval
  • Aug 16, 2010
  • View Source
  • When you subscribe to Account Data, there are about 102 fields of data sent by TWS. Some of them have very unusual names like "Indian Stock Haircut" or "lookAheadAvailableFunds."
  • Useful to your question are fields like "Buying Power" and "Available Funds."
  • Since I trade Futures, of interest to me are also "InitialMarginReq" and "MaintMarginReq."
  • For each contract I trade, my API looks at Available Funds minus my "Cusion" and Initial Margin Requirments for each contract, so I know how many contracts I can trade between, say, 9:30 and 3:45 EST.
  • After 3:45 EST, IB switches to Overnight Margin requirements, which are usually about double the Intraday requirements.
  • Can't help much with stocks and their margin requirements, but, like I said, there are about 102 fields of data, so I'm sure the information you're looking for is in one of them.
  • Good luck,
  • Leno
  • Show message history
  • Reply
  • Weidong
  • Aug 16, 2010
  • View Source
  • The DT sys is for stocks only, works between 9:30AM and 2PM.
  • The sys catched one of 102 fields after subscribing to Account Data, named "BuyingPower", as follows,
  • if(key=="BuyingPower")
  • {
  • m_BuyingPower = atof(val);
  • }
  • Before placing any buying order, sys check if m_BuyingPower minus shares*buyprice is bigger than 5000(always leaving 5000 dollars space), if so, go ahead to place order and minus shares*buyprice from m_BuyingPower. however still got error,
  • Id: 919 | Error Code: 201 | Error Msg: Order rejected – reason:YOUR ORDER IS NOT ACCEPTED.
  • IN ORDER TO OBTAIN THE DESIRED POSITION YOUR
  • EQUITY WITH LOAN VALUE [95269.89 USD]
  • MUST EXCEED
  • THE INITIAL MARGIN [95902.93 USD]
  • Anything wrong with this logic?
  • Thanks a lot,
  • Show message history
  • Reply
  • Kurt Bigler
  • Aug 17, 2010
  • View Source
  • Your logic "sounds fine" given a certain set of assumptions.
  • However I think the real question is how buying power relates to margin.
  • You say you have checked buying power and it is ok, and IB says they have
  • checked margin and it is not ok. So my superficial observation is that you
  • and IB are not talking the same language. I don't know offhand why (for
  • stock purchase) it would not work, except that I recall that some stocks are
  • not marginable, in which case the buying power notion would not apply.
  • Regardless, if you can compute it the way IB computes it (based on EWL and
  • margin) then maybe you will get the result you need, rather than trying to
  • second-guess how exactly how buying power relates to margin and when your
  • model breaks down.
  • But if indeed some stocks you trade are not marginable (penny stocks?) then
  • you would need to have a way to know that. Maybe use your buying power rule
  • for marginable stocks and use simple available funds for non-marginable
  • stocks.
  • -Kurt
  • Show message history
  • Reply
  • Weidong Ruan
  • Message 7 of 7 , Aug 17, 2010
  • View Source
  • Thanks Kurt, Really appreciated your invaluable input.

[Q] openOrder / orderStatus / execDetails sequence?

[A] by Kurt

  • Ron Hinchley
  • Message 1 of 2 , Jul 31, 2010
  • View Source
  • Does openOrder() always come before orderStatus() or execDetails() call backs?
  • Is the 'status' of openOrder() the same as the 'status' of orderStatus()?
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Message 2 of 2 , Jul 31, 2010
  • View Source
  • On 7/31/10 7:22 PM, "Ron Hinchley" <grocthis@…> wrote:
  • > Does openOrder() always come before orderStatus() or execDetails() call backs?
  • >
  • > Is the 'status' of openOrder() the same as the 'status' of orderStatus()?
  • It is not documented that I know of, so if we depend on it we may be
  • disappointed.
  • But yes, that's all true in my experience, re openOrder and orderStatus. I
  • currently use the order status from the earlier message and ignore the other
  • one.
  • Re execDetails I am not sure. It is such a separate entity with its own
  • kinds of unreliability that I don't imagine I would make any assumptions I
  • didn't need to make. For example I would be prepared to handle an
  • unexpected execDetails and likewise the lack of an expected execDetails. So
  • for example after a timeout I would probably request executions, but I
  • haven't done that code yet.
  • I would try to model state in a way that does not depend on the order of
  • execution versus order messages. However I currently plan on assuming
  • paired openOrder and orderStatus messages, unless I discover or hear of any
  • exceptions.
  • -Kurt

[Q] Orders pacing?

[A]

What I remember from a past experiment is if you send orders to the TWS

socket too quickly then TWS will deliberately or otherwise wind up not

accepting an order. It was not related to application processing or

timing. If you send three orders to the socket without delays you will

wind up with orders vanishing. If this has changed then great, but I

wanted to check to see what the current situation is.

From what I remember, and also from Joe's message, TWS does have some

issues if the orders arrive at the socket too quickly.

Measure time to the acknowledgement. If it's a TWS "synthetic" order

emulated by their servers, you get a "pending submit" status if I remember

correctly. If it's sent to the exchange, a "sumitted" status. Typically at

least on a fast exchange like Globex, the delay would be less than 100

msecs. More typically in the range of 60-70 msecs or so for

acknowledgement. I've measured this many times and there is some

variability but that's the speed you can achieve with a properly tuned

system.

In an effort to share some of my trials and tribulations with order placement,

there has always been a problem with IB's order handling (at least since early

2008, up through late 2009) that caused a problem with placing parent/child

orders where a single parent order is placed along with a single set of

bracketed children. If all orders are placed one after another (without any

pause in between them), the problem will manifest itself (at some point).

Last year, I talked to the IB support team for some time about the problem, but

they were never able to fix it or reliably reproduce it. I've written test

programs that create orders just like my application (in a paper trading

account), in rapid succession, but I still have not been able to reproduce the

problem. I've just never been able to properly reproduce the problem in a test

scenario.

The problem did not manifest itself as a pacing violation, but rather, the stop

child order did not get fully committed through TWS and the "T"ransmit button on

TWS was left in an unpressed state. This (of course) seriously sucked, because

my position was unprotected (and it actually burned me a couple times). I ended

up adding in a bunch of code that analyzes all orders to detect if this scenario

actually happens, so that my application can cancel and recreate the

orders appropriately, but with a small delay (mentioned below), the problem

never happens.

I trade US Equities and my system trades an average of about 13 positions per

day (some days less, but some days, I've had more than 40 simultaneous

positions). When I am not using the delay between order placement, the problem

manifests itself about once every 2 or 3 days and I have found absolutely zero

rhyme or reason as to why. The way that I do it and I have never had a single

failure with this technique (with over 9,000 trades) is by creating the parent

order, delaying for 200 milliseconds and then creating the child bracket orders

right away (with—of course—- the final stop order in the bracket being the

one with transmit=true).

Jason Sapp

Here's my two cents:

When I first began coding my API for trading, I built a mimic-TWS order server and found that there were, indeed, some timing issues while communicating over the TCP socket when the order server and my API were running on the same PC.

I had to place time delays here and there in order to get things to work without hangups.

There weren't any problems, however, when I used two computers.

Now, with TWS running on PC-1, for example, and my API and charting programs running on PC-2, when I place a "Basket" order, in which multiple buy/sell orders for DIFFERENT symbols are fired off in rapid succession, without delays, order-execution and open-order data all come back from TWS and everything works fine. No orders are lost.

I do not, however, place other orders, like bracket orders, for the same symbol at the same time, nor do I use the same orderID when placing new orders. I'm not "there" yet.

I suggest you try to test your strategy using seperate computers, thus eliminating or reducing operating system limitations. Also, try fireing off "Basket" orders of different symbols and see if they work, then Bracket orders, etc.

Good luck,

Leno

Yes, I was experiencing the same exact problem as Jason was, where I would

submit a parent order and then a couple of child bracket orders with the last

one being set to transmit=true. I also was placing about 10-15 trades per day

and without a delay of 300ms between the orders, quite often some orders would

show up in TWS, but without the "T" transmitted button pressed.

I would also get a response back through the API that said something like "

Can't find order with ID ="" " I tried as well to set up tests to see if I

could consistently replicate the issue, but to no avail. Even today with the

300 ms delay I will still on the rare occasion encounter this issue.

–Rob Terpilowski

[Q] How to modify submitted order?

[A1] by Kurt

You simply call placeOrder again with the same Order structure. If the

order id (and client id) are the same it is recognized as a modify to the

existing order.

–Kurt

[A2] by Kurt

You just keep your Order structure around (or all the info needed to

recreate it) and call placeOrder again with the same orderId and it will be

interpreted as a request to modify. Not all fields can be modified. Price

fields generally can be changed, and limit orders changed to market. I'm

pretty sure quantity can also be changed, but I've never done it.

Be warned that modifying an order will typically incur a cancellation fee,

although if the order (or another on the same day) ultimately executes you

may get execution credits against the cancellation fees. But the deal on

execution credits used to be great, and last I looked was almost worthless.

The cancel/modify fee depends on the exchange. Some exchanges charge

nothing for these things, but a directed order will generally have a higher

commission (and possibly poorer execution).

-Kurt

[A3] by  btw12342001 on the bracket order modification:

I was wondering why modifying a leg of a bracket order by changing the price and re-submitting didn't work.  It would trade through the price with no fill.  I noticed in TWS there was an update button and the order status in my log was "pre-submitted".  Obviously the trick is to set transmit=true for those that were false.  I just tested and this works fine.

Maybe obvious, but it vexed me for a while.

[Q] error: The price does not conform to the minimum price variation for this contract

[A] by Ed

ES moves in increments of 0.25 so your stop price must be a multiple of

0.25. For example, setting stop of 1092.60 will cause the error you are

seeing. This commonly happens when the stop price is the result of an

indicator calculation. Rounding to the contract tick amount is typically

the solution.

— also —

[q] A quick question, for emini, does the stop price need to confirm the minimum price variation or just LMT price?

[a] Your order will be rejected if any of the prices are not an exact multiple of the tick size.

[Q] Sometimes reqOpenOrders will not return all open orders. wtf?!

[A] https://groups.io/g/twsapi/topic/4044893#21013

[Q] I have not able to retrieve the previous filled orders through the reqExecutions call

[A] by Kurt

  • Hi there!
  • I have not able to retrieve the previous filled orders through the reqExecutions call. I played with the ExecutionFilter by putting a date string into the m_time and then passed it into the call. Regardless of the parameters I have used, such as setting the client id to "-1", the call would return today's fills. I was not able event to get yesterday's fills. So I suspect that I am doing something wrong here.
  • I wonder if anyone has success in using reqExecutions() call to get previous fills? What value should I use in the ExecutionFilter?
  • Thanks in advance.
  • Reply
  • Kurt Bigler
  • May 26, 2010
  • View Source
  • reqExecutions is apparently filtered by the set of days you have checked in
  • the Trades window in TWS, regardless of what you specify in the API filter.
  • I believe in any case you are limited to the last week.
  • If you are using the Gateway then I think you have no way to specify the
  • filter and I believe someone reported it defaults to the current day only.
  • If it is like TWS in that regard the "current" day might be the day you
  • started it. I have found that with TWS running from the day before I get no
  • executions info until I uncheck yesterday and check today in the Trades
  • window.
  • You can find more details on some of this in the archives from the last
  • couple of months.
  • -Kurt
  • Show message history
  • Reply
  • event_trader
  • May 26, 2010
  • View Source
  • Hi Kurt,
  • Thanks for the quick reply. You are right, if I specify the "All" option in the "trades" page, and leave the page open, I can get all the trades in the "trades" page using the API regardless of the filter parameters. It seems like that the TWS trades page just pass along all the trades in that page to the API call. This is a bad design on the IB API part.
  • I was hoping that the filter can behave more like the flex statement in the account management.
  • Show message history
  • Reply
  • Kurt Bigler
  • Message 4 of 4 , May 26, 2010
  • View Source
  • Thanks, I forgot it was necessary to leave the Trades window open until I
  • tested it just now. I suppose when leaving TWS running overnight and then
  • only getting yesterday's trades it probably means I *had* left the Trades
  • window open. Otherwise perhaps (to be determined) the Trades window will
  • always open with today checked even if it was started on a previous day.
  • Yes, this is definitely a serious problem for people who want to reconcile
  • everything. Also the downloadable statements do not (at least did not used
  • to) use the same form of execution id used by the API, which makes you have
  • to correlate based on the contract description, which I think will be a bit
  • more of a pain programmatically. I haven't approached the problem of
  • reconciling yet because as simple as it should be in concept these little
  • glitches make it a chore.
  • -Kurt

[Q] How to submit a vwap order with start and stop times and other options?

[A] by rfcd453

Since it is not in the current docs, here is how to do it (Java):

Contract m_contract = new Contract();

Order m_order = new Order();

Vector m_algoParams = new Vector();

TagValue m_tagvalue1 = new TagValue();

TagValue m_tagvalue2 = new TagValue();

TagValue m_tagvalue3 = new TagValue();

TagValue m_tagvalue4 = new TagValue();

TagValue m_tagvalue5 = new TagValue();

m_tagvalue1.m_tag = "maxPctVol";

m_tagvalue1.m_value = ".01";

m_tagvalue2.m_tag = "startTime";

m_tagvalue2.m_value = "14:00:00 EST";

m_tagvalue3.m_tag = "endTime";

m_tagvalue3.m_value = "14:30:00 EST";

m_tagvalue4.m_tag = "allowPastEndTime";

m_tagvalue4.m_value = "1";

m_tagvalue5.m_tag = "noTakeLiq";

m_tagvalue5.m_value = "1";

m_algoParams.add(m_tagvalue1);

m_algoParams.add(m_tagvalue2);

m_algoParams.add(m_tagvalue3);

m_algoParams.add(m_tagvalue4);

m_algoParams.add(m_tagvalue5);

m_contract.m_symbol = "AAPL";

m_contract.m_secType = "STK";

m_contract.m_exchange = "SMART";

m_contract.m_currency = "USD";

m_order.m_action = "SELL";

m_order.m_totalQuantity = 100;

m_order.m_orderType = "MKT";

m_order.m_algoStrategy = "Vwap";

m_order.m_algoParams = m_algoParams;

m_client.placeOrder(orderID, m_contract, m_order);’

[Q] Relative Order transmit problem

[A] basically there was no answer, but here’s the port (and definition of Relative Order):

Weidong Ruan

Jul 22, 2010

View Source

Hi,

Relative order sent from my application exists on TWS, but don't transmit.

If clicking T for this order on TWS, it go through without any problem.

Market order, limit order works well with my application. any problem in

following function? any suggestion are appreciated, Thanks.

void CClient2Dlg::placeRelativeOrder(CString symbol, int iOrderID, int

orderquantity)

{

Contract contract;

contract.conId = 1;

contract.symbol = symbol;

contract.exchange="SMART";

contract.secType = "STK";

contract.currency="USD";

Order order;

order.totalQuantity = orderquantity;

order.orderType = "REL";

order.action = "BUY";

order.lmtPrice = 0;

order.auxPrice = 0.01;

order.transmit = true;

m_pClient->placeOrder( iOrderID, contract, order);

}

Relative/Pegged-to-Primary Orders

(quote)

Relative (a.k.a. Pegged-to-Primary) orders provide a means for traders to seek a more aggressive price than the National Best Bid and Offer (NBBO). By acting as liquidity providers, and placing more aggressive bids and offers than the current best bids and offers, traders increase their odds of filling their order. Quotes are automatically adjusted as the markets move, to remain aggressive. For a buy order, your bid is pegged to the NBB by a more aggressive offset, and if the NBB moves up, your bid will also move up. If the NBB moves down, there will be no adjustment because your bid will become even more aggressive and execute. For sales, your offer is pegged to the NBO by a more aggressive offset, and if the NBO moves down, your offer will also move down. If the NBO moves up, there will be no adjustment because your offer will become more aggressive and execute. In addition to the offset, you can define an absolute cap, which works like a limit price, and will prevent your order from being executed above or below a specified level.

If you submit a relative order with a percentage offset, you are instructing us to calculate an order price that is consistent with the offset, but that also complies with applicable tick increments. Therefore we will calculate the order price rounded to the appropriate tick increment (e.g., pennies for a U.S. stock trading at a price over $1.00). Buy orders will be rounded down to the nearest acceptable tick increment and sell orders will be rounded up.

(end of quote)


Bracket orders

Definition of the bracket order (by IB)

[Q] Bracket order: Why Sell and Stop don’t cancel each other?

Hi,

I'm trying to place a bracket order (Buy, Sell LMT, STP) using the Java IB

API but the Sell and STP don't cancel each other.

I'm using the Buy order ID as the parent ID for both the SELL and STP

orders.

Any ideas what the problem is?

Thanx,

Rebecca.

Hi,

Here's the code:

Order order = createOrder("BUY", 1, "MKT");

Contract buyContract = createContract("ES", "FUT", "GLOBEX", "USD","20100618",null,0.0);

eClientSocket.placeOrder(orderId++, buyContract, order);

lastOrderId = orderId;

           

Order order2 = createExOrder("SELL", 1, "STP", lastOrderId, 0.0, currentPrice-gain, false);

Order order3 = createExOrder("SELL", 1, "LMT", lastOrderId, currentPrice+loss, 0.0, true);

           

eClientSocket.placeOrder(this.orderId++, buyContract, order2);

eClientSocket.placeOrder(this.orderId++, buyContract, order3);

Where the methods are defined below:

        protected Order createExOrder(String action, int quantity, String orderType, int parentId, double lmtPrice, double auxPrice, boolean transmit)

        {

           Order order = super.createOrder(action, quantity, orderType);

            order.m_parentId = parentId;

            order.m_lmtPrice = lmtPrice;

            order.m_auxPrice = auxPrice;

            order.m_transmit = transmit;

            return order;

        }

        protected Order createOrder(String action, int quantity, String orderType) {

            Order order = new Order();

            order.m_action = action;

            order.m_totalQuantity = quantity;

            order.m_orderType = orderType;

            // order.m_transmit = true;

            return order;

        }

        protected Contract createContract(String symbol, String securityType, String exchange, String currency) {

            return createContract(symbol, securityType, exchange, currency, null, null, 0.0);

        }

   

protected Contract createContract(String symbol, String securityType, String exchange, String currency, String expiry, String right, double strike) {

            Contract contract = new Contract();

            contract.m_symbol = symbol;

            contract.m_secType = securityType;

            contract.m_exchange = exchange;

            contract.m_currency = currency;

            if (expiry != null) {

                contract.m_expiry = expiry;

            }

            if (strike != 0.0) {

                contract.m_strike = strike;

            }

            if (right != null) {

                contract.m_right = right;

            }

            return contract;

        }

The result is that the Future was bought, and I had 2 open orders one SELL LMT and one SELL STP, that both seemed to have been transmitted.

Later on the STP caught and was executed, but the SELL LMT Order was not automatically cancelled.

Any ideas what I'm doing wrong?

Thanx in advance for the help,

Rebecca

[A1] by Jiang LingyuMay 31, 2010

you can set a same group name for your LMT and STP orders. This will cause OCO

[Rebecca]

I tried that, but it didn't seem to have any affect.

Can you give an example?

[A2] by Richard King

At a glance, I'd say that the problem is with your first call to placeOrder incrementing orderId. The increment happens after the current value is passed as the argument to placeOrder. Consequently lastOrderId is one greater than your entry order's id, and therefore the parentId of the child orders is not correctly set.

Richard

[A3] by Jiang Lingyu

From: TWSAPI@yahoogroups.com [mailto:TWSAPI@yahoogroups.com] On Behalf Of Jiang Lingyu

Sent: 01 June 2010 09:36

To: TWSAPI@yahoogroups.com

Subject: Re: [TWS API] bracket orders

Something like this:

order2.m_ocaGroup = "group1";

order3.m_ocaGroup = "group1";

hope it helps.

[A4] by Richard L King

Actually, TWS automatically assigns an OCA group to orders that have parentId set.

In fact it ignores anything you specify as in your example. (At least that used to be the case – I haven't tested this with recent versions.)

[Rebecca] (accepted answer:)

Thanx, that solved the problem.

I was looking for wrong usages of the API, instead of regular programming bugs πŸ™‚

I had another issue.

The STP was not transmitted even though the last order, the LMT sets transmit to true.

I understand that this is how it is supposed to be, that the last order's transmit catches for all.

It only worked once I set the transmit flag to true for all orders.

[note] by scourt2000

Also, on a market order, your distance from the two bracketing orders you setup will be off by a tick or two since you're entering them at the same time as your entry, especially with a market entry order. If you use a limit order, then the worst that will happen is that, in a volatile market, you'll get a better entry by a a tick and then you'll have a tighter stop-out than you originally wanted.

What I do is submit either a market or limit order and then I submit the bracket once I know the entry price as reported by IB.

[note]

All I know is that the brackets should have a matching, OCA

string. Another bracket needs a different string.

[note]

Let me spell it out again. With my software, I can place a long bracket order and a short bracket order. Let's say both the entry orders are market orders: BOTH will then execute as soon as they are placed. So as far as the IB account is concerned, the net position is now flat. But that has absolutely no effect on the two stop loss orders and the two target orders, which remain in the system (as two separate OCA groups) and will execute (and cancel the respective other halfs) when/if the relevant prices are hit.

If you have tried this and it didn't work, then you have simply done something wrong. You cannot tell me that it doesn't work, because it does!

Richard

[A5] by Ed (from other thread)

You need to set Transmit to False for all orders except the last one.

Send the Parent order first, then the children. On the last child order

set Transmit to True and the whole lot will be submitted.

— also good note from Ed —

I'm not sure what you are trying to achieve, but there are at least

two ideas:

1) Use the standard parent/child setup and adjust the child prices after

the parent executes

2) Submit a normal order for entry and then use an OCA group for the

profit/stop orders after the entry executes

[Q] Bracket orders: 3 or more childs? grand child? stop order as an entry?

I am newbie and wish you guys give me some advise.

Searched from API Reference Guide, I know there are 3 orders (one parent and two children) in a bracket order,

entry order (parent), market or limit, transmit = false

  stop order (child1), stop, transmit = false

  target order (child2), limit, transmit = true

Can we use stop order as an entry order?

Can we only have one child? Is it possible to add 3rd child order or more?

Is it possible to add a grand child order whose parent is child1?

Thanks

[A1] by Richard L King (from this thread)

My advice to you would be to find out answers to this sort of question yourself.

The amount of code needed to produce a small test app is not great, and producing it would be a good exercise for you. Once you have a basic working framework (connecting, logging errors, etc) you can easily clone it for new test apps in a few seconds.

Alternatively you could use the sample apps provided by IB to try these out – I very much recommend that you learn how to use one of these programs, because they can save you hours of time and you get the correct answer, whereas posting here you have a potentially long delay before you receive an answer, and there is no guarantee that any answer provided will be correct.

Having said that, I can tell you that:

– You can use a stop order for entry

– As far as I’m aware, you can use any number of child orders. Make sure you set transmit-false on all except the last to be placed. Note also that you can add child orders after the initial bracket order (or single order) has been placed

– I don’t know if you can use a child order as a parent, but I don’t see why not

But I urge you not to just believe me about any of this: try it yourself!

Richard

[A2] by orionn2 (from this same thread)

The answer to all your four questions is: yes, but to avoid problems place the stop loss last with transmit = true; parent and target should be placed before the stop loss order with transmit = false (this is to avoid entering a position without the stop loss order that may fail to be placed/transmitted due to an error – if the stop loss cannot be placed for some reason then the other orders will not be transmitted to the exchange).

[Q] Bracket orders: edge cases

For anyone using bracket orders(just saw another email asking about coding

example), could you answer my questions below?

1. If the parent order is a limit order and got partially filled, will the

two children still be activated? Or only when the parent got fully filled?

2. The two children work as OCA orders, right? Still about partial fill, if

one child got partially filled, will IB adjust the quality of another?

3. If the parent got canceled, will IB cancel children automatically? What

if the parent got partially filled then canceled?

Since I am using limit entry orders, the partial fill happens a lot, I am

wondering whether to use bracket orders could cover all the edge cases? If

not, I would rather handle by my own.

Thanks!

Jian

[A] by Christian Gross (from this tread)

It has been a little while since I did brackets. But here goes, anybody feel free to correct me.

1) yes

2) depends, by default yes, but check stock (assuming stock) OCA Group presets

3) yes the children will get cancelled. Assuming you did the cancelling. THOUGH there are exceptions to the rule. If there is a partial fill, and you cancelled just as the order was filled then your cancel will not be listened to. You get an error of non-existent order.

[Q] Bracket orders: How to place bracket order via API (recap)

Added on 28-Apr-2015

[A] by Richard L King (found here)

How to create a bracket order using the API has been discussed many times

here, and it certainly is possible (and very easy).

Here's how to do it:

1. Create your entry order, set transmit to false, and place the order.

2. Create your stop loss order, set its parentId to the entry order's order

id, keep transmit set to false, and place the order.

3. Create your target order, set its parentId to the entry order's order id,

set transmit to true, and place the order.

The transmit settings ensure that none of the orders are sent from TWS to

the IB servers until all the orders have been set up – then they are all

sent together. You don't have to do this, but if you don't you face the

potential problem of the entry order being filled before the stop loss and

target orders have been sent.

You don't have to have both a stop loss order and a target order. And if you

do have both, it doesn't matter what order you create them in (but of course

you must create and place the entry order first).

Don't be tempted to set the OCA group on the stop loss and target orders: it

isn't necessary, because TWS does that automatically and in fact it will

ignore anything you set.

You can modify the members of a bracket individually in the same way as any

other order.

Richard

[Q] How to modify modify pricing on a bracket order?

[A] by Josh (found here)

Added on 10-May-2017

It would be very similar for a bracket order. You just call placeOrder individually for the order(s) in the bracket you'd like to modify.


Placing Orders (stocks)

[Q] How can I keep a reference to these manual trades when they pop up in openOrders if all the IDs are 0 at the time?

[A] by Josh View/Reply Online (#40989)

Added on 01-Nov-2018

You can use the Order Reference field to manually label orders. For instance if you right-click on a tab in TWS Classic View and choose Settings, an Order Reference field can be entered which will be the default for all orders created on that tab. This string is then accessible to the API in Order.OrderRef.

[Q] What "Order Inactive" really means?

[A] 

  • When I try to place an order for an ES future, I'm getting "Inactive" from openOrder and from orderStatus, then an error message that the order was rejected. But I never get an order status of "Cancelled".
  • Regardless of the reason of the rejection which is not the issue here, is it true that I cannot determine from openOrder and orderStatus alone whether the order has been definitely rejected or if it just has been inhibited for some reason and could possibly be accepted at a later time?
  • PS: I already placed this question in the IB forum, and got no answer.
  • Reply
  • Richard L King
  • Aug 13, 2010
  • View Source
  • You get this when there is something wrong about the order that could be put
  • right. It gives you the opportunity to fix it and then resubmit it.
  • Of course an automated program is probably not going to be able to fix it,
  • or it wouldn't have submitted it incorrectly in the first place, so it will
  • probably want to cancel the order. But the API is also used by many manual
  • trading clients, so it may be that the user has specified something wrong on
  • the order and can perhaps put it right.
  • 'Inactive' means exactly that: the order will sit there indefinitely in TWS
  • awaiting further instructions. It won't somehow become active again without
  • user or API-client intervention.
  • From: TWSAPI@yahoogroups.com [mailto:TWSAPI@yahoogroups.com] On Behalf Of
  • Alexander
  • Sent: 13 August 2010 12:09
  • To: TWSAPI@yahoogroups.com
  • Subject: [TWS API] Order Rejected not an order status ?
  • When I try to place an order for an ES future, I'm getting "Inactive" from
  • openOrder and from orderStatus, then an error message that the order was
  • rejected. But I never get an order status of "Cancelled".
  • Regardless of the reason of the rejection which is not the issue here, is it
  • true that I cannot determine from openOrder and orderStatus alone whether
  • the order has been definitely rejected or if it just has been inhibited for
  • some reason and could possibly be accepted at a later time?
  • PS: I already placed this question in the IB forum, and got no answer.
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Aug 13, 2010
  • View Source
  • Contrary to what Richard said about this, I have found that in some
  • situations orders will be "Inactive" after an error, but you are *not* able
  • to modify and resubmit. I believe you get a duplicate order id (not sure)
  • if you attempt it.
  • Indeed as you said, there is absolutely no way to know whether "Inactive"
  • means the one thing (correct and resubmit) or other (create a new order from
  • scratch). It is a clear bug, and I have just never bothered to report it.
  • I believe what I started working on (I forget the status of that
  • development) is to make distinctions based on the error code and to correct
  • the order status.
  • (Sometimes I just get discouraged about reporting bugs to IB and just try to
  • find my own fix. This seems to be a fairly common attitude that develops
  • for people, from what I've been hearing on this list.)
  • There is yet another kind of problem that can come up. An order can be made
  • Inactive due to an error, and resubmitting is accepted, but correcting the
  • erroneous field does not clear the error. This is the case for setting (or
  • failing to set) the account field for a single-account order in a FA
  • scenario. Apparently IB's code reads the account field on the initial call
  • to placeOrder and does not update it on subseuqent placeOrder calls. So if
  • the account code was omitted it can not be added later and the order must be
  • scrapped.
  • -Kurt
  • Show message history
  • Reply
  • Alexander
  • Aug 16, 2010
  • View Source
  • OK, Order Rejected apparently does not exist as a order status and we don't know exactly what "Order Inactive" really means.
  • This is what I have recieved upon issueing a ES Sep 2010 order without sufficient margin, in this chronological order:
  • openOrder(…) with status Inactive
  • orderStatus(…) with status Inactive
  • error(…) 204 order rejected
  • openOrder(…) with status Inactive
  • orderStatus(…) with status Inactive
  • The order does not hang in TWS but does not appear at all. (Why openOrder and orderStatus two times?)
  • This I have tested on a F&A simulation subaccount, because institutional multiaccounts are not available as simulation accounts.
  • For the time being, assuming that my program does not issue orders with missing mandatory information, I will treat orders returning a status of Inactive as rejected and virtually nonexistent.
  • Thank you Kurt and Richard for your inputs.
  • Alexander
  • Reply
  • Kurt Bigler
  • FWIW, taking a look at my code, I might as well complete the info I previously started. I have not yet seen error 204. I have 201 listed as being prefixed
  • Aug 16, 2010
  • Alexander
  • Aug 22, 2010
  • View Source
  • Error 204 was a typo. Should be 201. The reason for the rejection was insufficient buying power. And no, I would never try to resend the order with the same orderId, I'm always using a new orderId. The red order on TWS will disappear sooner or later.
  • It is good to know that an order which has status inactive will never activated itself at a later time, as Richard pointed out, so I can treat it as rejected.
  • If anyone ever had an order status of inactive WITHOUT an accompagning error 201 I would like to know of it.
  • Alexander
  • Show message history
  • Reply
  • Kurt Bigler
  • Aug 22, 2010
  • View Source
  • I definitely had inactive without an error 201. For me the "Rejected" which
  • resulted in "Inactive" rather than "Cancelled" was new and anomalous
  • unusual-case behavior that broke code previously working written a couple
  • years earlier.
  • 201 is specifically "rejected". I have had orders remain inactive due to
  • some error or warning that did not say "rejected". (But also I think IB
  • keeps changing this kind of behavior.) I'm pretty sure every order I
  • submitted with transmit false used to produce an order status message
  • "Inactive". But if I remember right such orders no longer result in any
  • order status message at all, so I recoded my app to set my own order status
  • code "Local" for this situation in which only the API app seems to know
  • about the order. In short it used to be the case that "Inactive"
  • corresponded to orders displayed as "Gray" in TWS, which included orders
  • never transmitted and orders that were "soft" rejected. I used to get order
  • status messages for these in all cases (including untransmitted), and now I
  • no longer do in some cases. It is sort of a pain in the ass because it
  • means I have no feedback from TWS that it has received my untransmitted
  • order. So if I send it again with transmit=true I have no sense in advance,
  • like I used to, that all the order fields were already checked and the
  • transmit would fly.
  • So this behavior sucks but I'm so disgusted with the garbage IB produces
  • that I just sag slightly further into resigned depression instead of flying
  • into a rage like I used to.
  • -Kurt
  • Show message history
  • Reply
  • Alexander
  • Message 8 of 8 , Sep 2, 2010
  • View Source
  • Reply from IB's API support:
  • Hello Alexander,
  • You are right . When order has been rejected, OrderStatus event would show "Inactive" status and Error event delivers rejection message with the detail description. I strongly recommend that you monitor both OrderStatus and Error events to determine if the order has been placed successfully or not.
  • Regards,
  • Raymund
  • IB API Support

[Q] How to send a limit order then if the order is unfilled after a certain amount of time have the order be cancelled and do market order?

Basically can this be done in just one API call or do I have to cancel the order in code and then resend order as market order ? (Q On Mon, Nov 18, 2013 at 6:38 PM, <qjohnny2000@yahoo.com> wrote:)

[A1] (A by Brant Hahn brhahn@gmail.com via yahoogroups.com )

on your initial limit order, set the following value:

m_goodTillDate;  // FORMAT: 20060505 08:00:00 {time zone}

then when you get the cancelled callback from TWS, submit a new market order

[A2] Rob Terpilowski <rterp99@yahoo.com>

You may also be able to change the order type from LMT to MKT after your time threshold has expired.

[Q] order.m_account = null ?!

by orionn2 <no_reply@yahoogroups.com>:

has anyone ever experienced the situation of order.m_account field being null when orders are received in the openOrder method ? this issue is not allowing filtering of orders by sub-account. looking for your feedback.

[A] by Kurt Bigler kkb@breathsense.com

I see that in C++ the field is being decoded by the default OPEN_ORDER processing (for version 9.67), so you should be good in that regard.

However for various reasons I decided it would be a good idea to store/archive my own order information.  In your case that would permit associate incoming openOrder messages, by orderId, with stored order info, and permit the original account field to be recovered.

I'm not sure if it should be necessary in your case.  I discovered reasons for doing it related to the fact that orders can be sent to TWS with placeOrder false, and such orders are not reported back by reqOpenOrder, etc., thus orphaning them from API access if my API app quits without saving its own order state and restoring it on startup.  Some things related to this also got worse at one point (in one of IB's improvements), forcing the issue.

-Kurt

[A2] by vanx23@outlook.com

This sounds suspicious to me.

I have two individual accounts. The field 'AccountID' is never NULL or empty for each OpenOrder messages my application process.

[A3] by orionn2 <no_reply@yahoogroups.com>

Just to clarify that this is happening on an Advisor account (not on individual accounts) when an order is being created and automatically split/allocated to three sub-accounts using the latest TWS version.

[Q] m_shortSaleSlot's field?

[A] by Kurt

The comment in C++ says:

"1 if you hold the shares, 2 if they will be delivered from elsewhere.

Only for Action="SSHORT"

And I believe (but can not find the doc saying it) that SSHORT is

institutional only. It makes sense anyway.

This does not specifically say what you should set the field to if it

doesn't apply. In such cases with TWS API, I let the default initialization

of field determine the value to use. shortSaleSlot is defaulted to 0 by the

C++ constructor. So just don't set the field if it doesn't apply, and the

default should be right.

-Kurt


Placing Orders (options)

[Q] How to get daily volume for options?

Options don't have end of day data so the best I could come up with was getting large bars (like 8 hours) and using those to see if there was volume?

[A] by  Josh View/Reply Online (#40025)

Added on 10-Jul-2018

That should work fine. There isn't really any other way to do it.

[Q] How to Monitor Stock Loan Availability?

[A] by  Jack Jost jqjost@gmail.com View/Reply Online (#39012) 

Added on 27-Nov-2017

(quote from https://ibkr.info/article/2024 )

Monitoring Stock Loan Availability

Overview:

IB provides a variety of methods to assist account holders engaged in short selling with monitoring inventory levels and borrow costs/rebates. The level of detail available, the time frame covered and the manner in which the information is accessed vary by method and a brief overview of each is provided below.

Public Website

Interested parties may query the public website for stock loan data with no user name or password required. To start, click here and select the country in which the stock is listed. If the number of available issues exceeds that which can be reasonably presented on a single page, results will be organized by symbol in groups, with hypertext links allowing further drill-down. A quick search box allowing direct query for a given symbol is also provided. Query results include the product description, currency of denomination and a link titled “Check Availability” which displays the quantity of shares available to borrow.

 

Public FTP

The public FTP site also requires no user name or password to access and provides stock borrow data in bulk form via a pipe delimited text file. The URL necessary to request files varies by browser type as outlined below:

1.        Mozilla Firefox

ftp://shortstock: @ftp3.interactivebrokers.com (Note there is a space between shortstock: and @ which represents the password).

You will get a message stating that you are about to connect to a website that does not require authentication. Select "Yes"

2.        Internet Explorer

ftp://shortstock: @ftp3.interactivebrokers.com (Note there is a space between shortstock: and @ which represents the password).

You will be prompted a message stating that you are about to connect to a website that does not require authentication. Select "Yes"

3.        Google Chrome

ftp://ftp3.interactivebrokers.com

Username:shortstock

Password: blank (n/a)

Within the site, individual files will be organized by country of listing with checkboxes provided to specify those desired which can then be downloaded into a single file by selecting the Submit button.

 

 

Outlined below is a snapshot of the sample file output which includes the stock symbol, currency of denomination, name, contract identifiers (IB’s and the ISIN), rebate & fee rates and shares available. This file may be also imported into applications such as Excel for sorting, filtering and analytical purposes.

 

Stock Loan Borrow (SLB) Tool

The SLB tool is made available to IB account holders through login to Account Management (Support then Tools menu options). This tool allows one to query information on a single stock as well as at a bulk level. Single stock searches can be performed by symbol/exchange, ISIN or CUSIP numbers. At the single security level, query results include the quantity available, number of lenders and indicative rebate rate (which if negative, infers a borrowing cost expressed as an annual percentage rate and, if positive, the interest rebate paid on cash proceeds securing the loan in excess of the minimum threshold). Information regarding the quantity of shares available to borrow throughout the day for the most current and past half hour increments is also made available.

 

In addition, borrowers interested in the trend of rates over the prior 10 day period can view the minimum, maximum and mean rates for each day.

This tool also allows one to upload a text file (with symbol/exchange or ISIN detail) and search for availability of multiple stocks in bulk within a single query. These bulk requests will then generate a .CSV file similar to the sample file output made available through the public FTP site.

Related Articles

(/quote)

[Q] How to find SLB info?

I'd like to get the SLB (Short stocks Loan / Borrow) fee rate via API.

I can't find it anywhere (tick data, fundamental data reports from Reuters…).

Anybody knows if it is possible?

[A] by mattpearson911@yahoo.com

Added on 07-Oct-2016

You can download them here:

ftp://ftp3.interactivebrokers.com/

login: shortstock

No password (leave blank)

[Q] TWS and Option Spreads

[A] read full  thread

[Q] weird theta

Has anyone seen a very large theta returned by tws api? I got the theta 1.7976931348623157E+308 for tsla 195C. It must be an error at tws server side, right? btw, I got all other greeks right.

[A] by Kurt

Are you sure that isn't the max double value used to mean undefined or no data?  It is a very familiar looking number, amazing how many times it comes up in questions on this forum.  Wikipedia shows:

    7fef ffff ffff ffff16   = (1 + (1 – 2-52)) × 21023 

                        ≈ 1.7976931348623157 × 10308 (Max Double)

IB is inconsistent in quotes as to what value is used for the no data case.  Sometimes it is –1, sometimes -.99, sometimes 0, sometimes the "maximum" value.

At the moment my app is displaying the undefined value for many TSLA calls expiring today.  My code converts the special values to something displayable.  I checked a couple other expirys and they have normal values.

IB's strategy is for the greek to be undefined if the calculations are not stable, which in the case of theta is not surprising on expiry day.

-Kurt

— also —

Ding,

[These comments apply to C++.]

The value in question is DBL_MAX, or that is one way you can refer to it in your code, and is one way IB refers to it in their socket interface layer code (e.g. in EClientSocketBase::EncodeFieldMax), although they also use

    #define UNSET_DOUBLE DBL_MAX

in some cases (e.g., EClientSocketBase::DecodeFieldMax, demonstrating some inconsistency).

It is not at all weird for the API to return DBL_MAX for a greek.  You will get that value all the time and you will need to check for DBL_MAX if you are using the greek values from the API.  I just checked my code and it is indeed DBL_MAX that I check for and interpret in the usual way (for the TWS API) as meaning no value is available.  I check for DBL_MAX in the case of the following arguments to the tickOptionComputation callback.

    impliedVol

    delta

    gamma

    vega

    theta

    optPrice

It may be true that these values occur in the API even when TWS is still displaying a value.  It might just be that TWS is a lot less fussy about displaying something (valid or not) whereas IB may not want to risk sending an unstable value to the API when it is conceivable that someone might conceivably trade automatically on it.

Come to think of it my recollection is that greeks are always displayed in the option trader.  I could be wrong but don't remember the columns ever being sparsely blank.  Meanwhile I am used to seeing blank greek columns in my API app, quite often (based on the occurrence of DBL_MAX), intermixed with other greeks for which values are provided.

I see that my code also checks UNSET_DOUBLE (same value as DBL_MAX) for commission and various order fields.  This value (or UNSET_INTEGER = INT_MAX in the case of integers) is used in a lot of places.

It had been my impression that this value may be used for almost any tick (quote) field but at the moment I am not finding evidence of that.  For TICK_PRICE/tickPrice, DECODE_FIELD is used, not DECODE_FIELD_MAX.  DECODE_FIELD_MAX appears to be used for OPEN_ORDER only.  DECODE_FIELD_MAX is actually only used to permit IB to send the null string on the socket and have it mean DBL_MAX.  DBL_MAX can always be sent on the socket directly, and apparently that is what is done for the greeks since DECODE_FIELD, not  DECODE_FIELD_MAX is used there.

However I see no signs of my code special-casing or DBL_MAX or UNSET_DOUBLE in my code except as mentioned above.

-Kurt

[Q] How should an API client detect an option exercise, the exercise being done manually in TWS?

Hi,

How should an API client detect an option exercise, the exercise being done manually in TWS?

The API client receives an open order event, but annoyingly enough, the order event is always  a "buy" instead of the more logical "sell" (an option exercise can only be done on a long option; closing a long position is better characterized as a sell than a buy, in my mind):

5;34;-279;283326534;DIA;OPT;20171020;231;C;100;SMART;USD;DIA   171020C00231000;DIA;BUY;100;LMT;0.0;0.0;DAY;;U###;C;0;;0;1689659455;0;0;0;;1689659455.0/U###/100;;;;;;;;;;0;;-1;0;;;;;;;0;0;0;;3;0;0;;0;0;;0;None;;0;;;;?;0;0;;0;0;;;;;;0;0;0;;;;;0;;IB;0;0;;0;0;Inactive;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;;;;;;0;0;0;None;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;0;;;;1.7976931348623157E308

If the order event was a sell, then it would be easy to handle just like any other order event received from TWS.

Alternatively, if the order event was somehow identified as an exercise, then I could swap the buy to a sell.

But as it is, I can't see anything in the order event that shows it as an exercise, and I can't just go swapping buy to sell in all order events I receive…

Thanks!

Jimmy

[A] by Josh: View/Reply Online (#38869)

Jimmy,

I agree its not the most intuitive, but option exercise instructions will always appear with order side = "BUY" to orderStatus. The order status behavior is also different because there will not be an orderStatus returned after calling reqOpenOrder or reqAllOpenOrder, only a warning message.

The exercise request can be identified by the '0' limit price, since this is not possible for any other orders not involving a combo contract.

Josh

[Q] How to find out available strike prices for options ?

happen to be the same as

[Q] How to pull entire option chain?

[A] by Kurt (from this thread)

There is no way to get *just* the strikes. To get all possible strikes

across all possible expiry's you have to poll the entire option chain and

take note of what you want and throw the rest away.

To do this you use reqContractDetails with an ambiguous contract

specification, i.e. out of strike, expiry, and right, you simply don't *set*

the fields you want wildcarded. Then you get back a whole series of

contractDetails responses from the single request, which you can track by

the request id. You have received the whole chain when the

contractDetailsEnd message arrives for that request id.

You can by this method also specify an expiry and right ("C" or "P") and get

all strikes for that expiry and right. Any combination you like. But there

is no independent list of expiry's and no independent list of strikes, only

the combinations. If you leave the right unset ("") you will get both calls

and puts.

When last tested, you can also partially wildcard the date, e.g. by

specifying only the beginning portion of the date, including just the year,

or just the year and the month (e.g. "201201") rather than specifying the

full date code. This is less convenient now with weekly expiry's available

(pain in the butt if you ask me). Too many expiry's, too many strikes.

(And somebody *else* gets to make out on the arbitrage.) Oh for the good

'ol days.

-Kurt

[A2] quote from this thread by Kurt

Try:

conId=0 symbol="AAPL" secType="OPT"

expiry="20110415" strike=345 right="C" multiplier=""

exchange="SMART" primaryExchange="" currency="USD" localSymbol=""

includeExpired=0

Or:

conId=0 symbol="AAPL" secType="OPT"

expiry="" strike=0 right="" multiplier=""

exchange="SMART" primaryExchange="" currency="USD"

localSymbol="AAPL 110416C00345000"

includeExpired=0

Note the two different variants on the date for expiry versus the OSI

symbol.

With futures I believe someone here said you can omit the symbol if you

specify the contract fully by localSymbol, but I've never tried it for OSI

options.

While we're at it I might as well tell you a couple other things you may run

into:

* Option chains are requested via reqContractDetails with some fields (e.g.

strike, right, and/or expiry) omitted to serve as wildcard. There is no way

to just request a list of expiries or strikes, only a full list of matching

contracts.

* You might get more reliable results in some cases (avoiding certain kinds

of errors) if you obtain a conId via reqContractDetails and use that conId

*instead of* the other fields to specify a contract for reqMarketData and

reqHistoricalData, and perhaps most importantly, placeOrder. This may be

less true now than in the past but it used to be that placeOrder was more

fussy than the other requests and would reject contracts that the other

requests accepted. Probably it is still true that reqContractDetails is

less fussy than the others due to the possibility of wildcarding, which come

to think of it (speculating) might apply even if there is only one match.

-Kurt

[A3] by dthayer

Cut and paste of code I use to get option data. If set strike=0.0 and expiry=0.0, and right="", then you get the entire chain for both puts and calls.

        Contract *C = new Contract();

        C->symbol        = "KMB";

        C->currency        = "USD";

        C->secType = "OPT";

        C->expiry = "20130621";    

        C->strike = 95.0;                        // Define as 0.0 to get entire option chain

        C->right = "P";                                // Put no string, 2 double quotes, if want both put and call

        C->exchange        = "SMART";

        C->multiplier = "";

        C->primaryExchange = "";

        C->localSymbol = "";

        C->includeExpired = 0;

        m_pClient1->reqMktData(msgID,

C,"233",false);        // false – start stream, get tick types

                                                                        // Wait in TWS::contractDetails for reply from TWS

        //m_pClient1->reqMktData( msgID,C,"",true);        // true – get snapshot, don't specify tick types.

[A4] by rholowczak

Here is a quick tutorial how to get the contract details for all options on an underlying stock (option chains) using a simple Java program and the IB Java API: Programming Interactive Brokers Java API in a Console Application – Contract Details

[Q] Why option chain data is delayed (sometimes up to 1 minute)?

[A] by IB support: (thanks to joe@higgs-tx.com)

Added on 18-Apr-2016

Unfortunately this is by design a pacing limitation to prevent clients from putting to much stress on our servers. When requesting contract details for an option chain the contractDetails () call back method will be buffered for each additional request. Below is a breakdown example on the contractDetails buffering.

For Examples:

*************************************

If Expiry = Blank or NULL

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to null or empty string, the delay is 1 minute.

*************************************

*************************************

If Expiry = YYYY (e.g. 2016)

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “2016” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year only, the delay is 1 minute.

*************************************

*************************************

If Expiry = YYYYMM (e.g. 201603)

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “201603” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year and month format (YYYYMM), the delay is 5 seconds.

*************************************

*************************************

If Expiry = YYYYMMDD (e.g. 20160318)

Symbol = “IBKR”;

secType = “OPT”;

Expiry = "20160318"

Strike = "31";

Right = "C";

Multiplier = "100";

/TradingClass = "IBKR";

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year with month plus the day (YYYYMMDD) AND Strike, AND Right, AND Multiplier or TradingClass included then there is NO delay.

Please note, that this ONLY works with front month option contracts and ONLY for the first 8 requests. If you are not requesting the front month and requesting more than 8 requests then it defaults back to a delay of 1 minute.

—- Also from jb201448@yahoo.com  —

There's also a new function reqSecDefOptParams() in the Beta version which should return option chains faster.

-Josh

[Q] Is IB’s model reliable or useless and I need to do all the calculations myself?

[A]

see this thread: https://groups.io/g/twsapi/topic/4045086#21842

[Q] Am I required to pay the dividends on shares borrowed when short selling?

[A] by Frank Bell

Dividend obligations only occur if a position is held at market close the day

before the ex-dividend date. That's true for spin-offs and all other corporate

events.

[Q] Which stocks are shortable which are not?

[A] by Jason

For a list of

shortable stocks with IB (for all countries), you can go here:

http://www.interactivebrokers.com/en/p.php?f=shortableStocks. For a complete

list of shortable US stocks, you can go here:

http://www.interactivebrokers.com/en/trading/ViewShortableStocks.php?cntry=usa&tag=United%20States&ib_entity=llc&ln=.

Lastly, for a list of US stocks that are not shortable for regulatory reasons,

you can go here:

http://www.interactivebrokers.com/en/trading/ViewShortableStocks.php?cntry=regSHO&tag=Reg%20SHO&ib_entity=llc&ln= (although

some stocks on this list may still be shortable via IB).

[Q] Unable to borrow?

[A] by Frank Bell

You can repeatedly reduce your size on unable to borrow orders until the order

is successful or you reach your minimum order size. Also, be aware that open

short orders can go from submitted to pre-submitted when you lose your borrow.

[Q] How to determine the pricing step for an option?

Hi,

Some of the options are priced with gaps, for example, if the pricing  step is 5

cents and you submit an order to buy an option for $1.02, your order will be

rejected, because the valid priced would be $1.00 or $1.05

Now, is there a way to determine the valid prices programmatically, any code or

formula (the programming language doesn't matter)

Best Regards

– JW

[A] by Kurt

Read about the minTick field of the ContractDetails structure.

-Kurt

[Q] placeOrder() No security definition has been found for the request

[A] see [Q] Errorcode(200) No security definition has been found for the request

[Q] I see duplicate events from time to time – wtf?!

[A] From IB's docs:

Note:  It is possible that orderStatus() may return duplicate messages. It is essential that you filter the message accordingly.

[A2] by Kurt

But I don't think there is anything wrong with duplicate events in this case.  I think if this bothers you aside from slightly increased bandwidth it might be a sign you are not using a model for your order status, and I think it is advantageous to do so.  You simply need to update your model from the status most recently received, and of course process all changes in status as required by your system.  It is after all very easy to detect that a change has occurred relative to the previously received and stored status.  The order status is the cumulative result of all prior activity.  As such, if a duplicate order status causes you a problem, there should be a trivial change to your algorithms that should eliminate any sensitivity to it.  It is good practice to do that anyway.  

[A3]

Also interesting that they don't send duplicate execution events, just duplicate order status events.

[Q] attached order

[A]

You should set .transmit=0 for the entry order and .transmit=1 for the

attached order.

If you set .transmit=1 for both, then TWS submits the entry order before it

knows anything about the attached order, and it may well execute before the

attached order arrives at TWS, in which case the error you described will

occur.


[Q] Bad equity options spread (bid-ask seems to widen significantly)

[A] read the whole thread


Placing orders (futures)

[Q] Can I use SMART for futures?

[A] rwk2095

SMART routing does not apply to futures.  You have to specify the exchange, and there is no extra commission for that.

[Q] Can I place MOC order for es future?

[A] by Rob Terpilowski

MOC orders do not exist for futures on any exchanges that I know of..

[A2] by Frank Bell

MOC isn't supported for ES. What I do is place a market order with a good after

time of 16:14:58. You could probably set the time at 16:14:59, but I like to

give IB a second of grace.

Frank

[Q] futures options data via API

[A] by souqmate (found here)

Here's a roadmap for fighting your way through IB's jungle of options on US futures (FOP).

FOP historical data are not available for expired contracts (if s.o. has recorded older data, I'd be glad to trade).

Weekly options are always European-style; they exist for forex (AUD CAD CHF EUR JPY), rates (ZT ZF ZN ZB), indices (ES NQ SPX).

Forex: have monthly American/European options: 6A/XA 6B/XB 6C/XD 6E/XT 6J/XJ 6S/XS  (actually: no XA; IB forgot?).

NG: have classes ON/LNE for monthly American/European options (but no feed nor hist data for LNE – though 4* more liquid).

ES: have weeklies (EW1 EW2 EW4), monthlies (ES in place of EW3) and EoM (EW, expire on last day of month).

NQ: have weeklies (QN1 QN2 QN4), monthlies (NQ), and EoM (QNE).

SPX (fut): have weeklies (EV1 EV2 EV4), monthlies (SP), and EoM (EV).

ZC ZS (not ZW): have usual monthlies (OZC OZS) and also monthlies whose underlying expires in Dec (OCD OSD), both American-style.

Eurodollar options (GE@Globex): GE GE0 GE2 GE3 GE4 (Globex), ED E0 E2 E3 E4 (CME).  Have no weeklies, only monthlies.  All underlying futures have (quaterly) expiries in same year (GE) or next years (GE0 GE2 GE3 GE4); all American-style.  Dunno why GE0 is not rather called GE1.  

Expiry date for GE can differ from that of GE0…GE4 for Dec month, eg 20131216 vs 20131213.

minTick can depend on expiry, eg: on 20131011, GE has minTick=0.005 for 201406 and 0.0025 for 201403.

strikeIncrement can depend on expiry, eg ZM: first expiry has increments of 5, the second of 10.

strikeIncrement can be smaller atm and bigger away: GF has strikeIncrement=0.5 close to atm and 1 away from atm; ED has strikeIncrement=0.125 close to atm and 0.25 away from atm.

SI: actual strikeIncrement is 0.05 around the money, but only 0.25 seems used.

ZC ZL ZS ZW GF LE HE (but not ZM): have factor 1/100 in strikes; only in API, not in TWS; IB won't change that.

No FOP at IB for NKD PA PL VIX.

localSymbol has no standardised syntax, eg: "6AZ3 P1075" for AUD@GLOBEX, but "C OYM  SEP 14  11600" for YM@ECBOT.

Three ways to specify contracts in queries for reqMktData or reqHistoricalData:

———————————-

symbol = GBP

secType = FOP

expiry = 20131108

strike = 1.55

right = P

tradingClass = 6B

exchange = GLOBEX

————————————-

currency = USD

conId = 132707825

exchange = GLOBEX

————————————-

localSymbol = 6BX3 P1550

secType = FOP

exchange = GLOBEX

————————————-

These work for API 9.69 or higher (since Jul 2013), where tradingClass has been pulled from ContractDetails to Contract.

For API 9.68 or earlier: only the first of these queries works; specifying conId is redundant, as strike,expiry,right are compulsory; hence for NG, forex and GE options: need to specify localSymbol when querying hist data for monthlies, to disambiguate between American/European styles (forex, NG) or midcurve options (GE0 GE2 GE3 GE4); conId is again useless here, and so are tradingClass and marketName.  

Annoying localSymbol syntax: have always 4 digits at end of localSymbol for forex  ("6SH4 P0950") or GE ("GEV3 C0112"), but 3 or 4 digits for NG ("ONX3 C900" or "ONX3 C1000"); the number is 1000*strike for forex & NG, but 1e5*strike for JPY.  For GE, have "GEV3 C9987" for strike=99.875, but "GEV3 C0112" for strike=101.125; so do printf("%04d\n",100*strike %1e4).  

However, since API 9.69, we can use tradingClass or conId instead of localSymbol to disambiguate.

Daily trade volume in IB files is ca. half of CME volume in ftp://ftp.cmegroup.com/pub/settle/stl*

I suppose the rest comes from the pit.

Happy FOP,

souqMate

[Q] How to get Futures Front Month

How do I ensure I'm always getting the active-month of futures contract?

I can either use includeExpired = True and check the date, but I would like something more robust.  Once it expires in the fourth week, I start getting this message: ERROR 1 200 No security definition has been found for the request and there's no callback to capture this.

Here's what I'm using:

        def subscribe(self):

            c = ibapi.contract.Contract()

            c.symbol = 'CL'

            c.secType = 'FUT'

            c.currency = 'USD'

            c.exchange = 'NYMEX'

            c.lastTradeDateOrContractMonth = "201706"

            reqId = self.getReqId()

            self._reqId2Contract[reqId] = c

            self.reqContractDetails(reqId, c)

I decided to use includeExpired=True and loop the month until I hit an unexpired contract

I thought there was a way to get a list of all futures expirations. Maybe I'm mistaken and that was for option chains

[A] by souqmate (found here)

Added on 15-May-2017

reqContractDetails() with

 symbol="CL" secType="FUT" exchange="NYMEX" currency="USD" includeExpired="1"

fires back 97 contracts, of which 24 expired (IB keeps 2 yrs worth of futures data – then discards them).

If you want the front month, drop includeExpired and find the youngest of the 73 contracts in the output; that won't be necessarily the most traded one.

I advise you to have your own calandar of roll dates, eg.here

[A2] by Nick

If you are going to be trading (not just academic research) almost all activity moves to the next contract before the current contract expires. This is the rollover date that souqmate advises you to be aware of and I agree with him.

[A3] by Mark

Hi Jared,

I'm new to the IB API so there's almost certainly a better way than this, but my solution was to calculate the expiry dates myself and switch to the next future as it expired.

Note that no actual harm come from requesting a wrong future, so you can suppress the error and spread your requests.

Best wishes,

Mark

[A4] by J G

If you do not specify lastTradeDateOrContractMonth you will receive information on all futures contracts.

Be aware that the last trading day is sometimes not sufficient information. For certain contracts with physical delivery you need to be out of the position before the First Position Date. See here for more details -> http://ibkb.interactivebrokers.com/node/992.

[Q] continuous contracts rollover schedule

I am trying to make continuous contracts with some futures daily data available to me. In my code, I plan to roll over to next contract based on specific dates(e.g. 1st notice day, or certain days before expiry) rather then open interest or volume. However as the rollover rules are different for different future contracts,i was wondering if anybody can help me find where to get the continuous contract rollover schedule for different futures. I have tried searching over Google and the group conversations archive but have not been successful in finding an answer. I would be much obliged if somebody can guide me.

regards,

Fatima.

[A] by souqmate (found here on old yahoo groups, now moved to here on groups.io)

below are my calendar rules for rolls (I suppose they vary from one person to the other).

They use Reuters rics, so I'll let you convert to IB symbols where needed.

When I say roll on day D, that means the new contract is used to trade on the morning of day D.

You'll find expiry dates in the output of reqContractDetails.

Gd luck

souqMate.

STXE FDX FSMI FFI IFS IFX, only HMUZ months, roll on day of expiry:

FCE MFXI OMXS30 AEX, each month, roll on day of expiry:

ES NQ YM DM TFS RMF, only HMUZ, roll on Friday prior expiry (ie 5 days earlier – Thu for YM):

CL NG, all months, roll 1 day prior expiry day:

HO RB, all months, roll 8 days prior expiry day:

GC, only GJMQZ, roll on 3rd to last day prior expiry month (was 2nd prior IBprod):

SI HG, only HKNUZ, roll on 3rd to last day prior expiry month (was 2nd prior IBprod):

PL, only FJNV, roll on 3. to last day prior expiry month:

PA, only HMUZ, roll on 2. to last day of month prior expiry month:

FGBX FGBL FGBM FGBS NK JNI JNM YAP SXF, only HMUZ, roll 1 day prior expiry day:

ZB ZN ZF ZT, only HMUZ, roll on 2nd to last day prior expiry month

FSS FEI FES JEY ED BAX YBA, only HMUZ, roll on day after expiry and always use c2; eg H7 expires 20070321, so use M7 till 20070321 and use Z7 since 20070322.

FLG CGB, only HMUZ, roll on 3rd day prior end of month preceding expiry:

YTC YTT JGB, only HMUZ, roll on day of expiry:

IND, only GJMQVZ, roll on day of expiry:

CME agric (ZC ZW ZS ZL ZM LC FC LH): roll in last week of month prior expiry month;

ZC/C: HKNUZ, skip all U contracts (except U05 20050629-20050713);

ZW/W: HKNUZ, skip most U and many K contracts;

ZS/S: FHKNQUX, use only FHKNX except Q03 and U03 (instead of X03);

ZL/BO: FHKNQUVZ, use only FHKNZ (and Q till Q04);

ZM/SM: FHKNQUVZ, use only FHKNQZ (skip Q sometimes);

LC: GJMQVZ,

FC: FHJKQUVX, switch using RDTH volumes (discarded 3 U-contracts); can skip U contracts

LH: GJKMNQVZ; use only GJMNQVZ; switch using RDTH volumes; roll on 15.-25. of month prior expiry month.

ICE futures:

CC: HKNUZ; roll in 2. week of month prior expiry month;

KC: HKNUZ; roll in 3. week of month prior expiry month;

CT: HKNVZ but use only HKNZ; roll around 15. of month prior expiry month;

SB: HKNV, roll in 3. week of month prior expiry month;

LCO/CO: all 12m FGHJKMNQUVXZ; roll 2d prior expiry  (like CL which rolls 1d prior expiry);

CME forex: URO BP JY RP SF CD AD KRW MP DX, all HMUZ (but all 12m for KRW – on Kor Fut Exch); roll  1 day prior expiry; last trade at 9:16 a.m. (Chicago time) on the 2nd business day preceding the third Wednesday of the contract month (usually Monday);

——————————————–

Souqmate,

Useful information, thanks. Do you also perform back propagation of price adjustments when you roll? To create a true continuous contract historic data file one would need to adjust prior prices at each contract roll no? Rather like back adjusting for splits for equities.

————————————————-

Yes, I roll by difference, using the difference of the settement prices (or last trade prices prior settlement time on eve of roll date) between the old and new contracts. That gives me a continues series for computing PnL in strategies.

souqMate.

[Q] Continuous contract – how to determine offset?

[A] https://groups.io/g/twsapi/topic/4044951

[Q] How to create a true continuous contract historic data (futures)? Do you also perform back propagation of price adjustments when you roll?

Souqmate,

Useful information, thanks. Do you also perform back propagation of price adjustments when you roll? To create a true continuous contract historic data file one would need to adjust prior prices at each contract roll no? Rather like back adjusting for splits for equities.

[A] by souqMate found here

Added on 19-Aug-2016

Yes, I roll by difference, using the difference of the settement prices (or last trade prices prior settlement time on eve of roll date) between the old and new contracts. That gives me a continues series for computing PnL in strategies.

souqMate.

[Q] Futures: what is “First Position Date”?

[A] by Malcolm Haylock

Added on 20-Aug-2016

Thanks for the roll schedule. Having to maintain this is obviously sensitive to changes in the contracts.

The first position date is part of the contract. Open the contract details from the IB website and you’ll see it:

https://pennies.interactivebrokers.com/cstools/contract_info/v3.9/index.php

For many contracts the liquidation date depends on the first position date:

http://ibkb.interactivebrokers.com/node/992

If there was a way to retrieve through the API the first position day then it would be much simpler to create a rollover strategy.


Combo Orders

[Q] Placing Combo orders?

[A1] (about troubles with placing combo order using paper account)

Found here: View/Reply Online (#197)

Added on 29-Aug-2017

In case somebody runs into the same issue, I talked to IB support.  They escalated me to the Trade Support Team who said that this is a known problem with combo orders in paper accounts.  Combo orders are buggy in paper accounts but they assured me that it would work fine with a regular account.  There is no known workaround to auto-transmit the combo orders other than doing it via the TWS user interface.

[A2]

has to get the "conId" thru "reqContractDetails" then add up the legs thru "addComboLeg".

todo: add an example here

[A3] by Kurt

what I do is create an object for each leg and an object for the

combo. I request contract details for each leg, with a different request

id. Then when contract details results come back the request id tell me

which leg the results are associated with. The combo object is notified

when the legs have received their contract details data. When data for all

legs has been received, then the combo object is complete, and at that point

I create the Contract object for it, with the legs, and immediately request

market data for the combo. That will both give me quotes for the combo and

also validate the combo Contract object. Or an error might come back from

the market data request telling me (error 200) "No security definition has

been found for the request", in which case I know that IB does not support

that combo or no one is providing liquidity for it (or else that I have a

bug).

One thing you may be missing is that besides the info for the legs, you must

specify the underlying symbol for one of the legs as the symbol for the

combo itself. This is a little piece of nonsense in my opinion but IB now

requires it. It is mentioned as an API topic in one of the more recent

release notes for TWS itself (not the API).

-Kurt

[A3] by fantasyfootball672

Hugh, I have a structure in here with some of the "fill"- you can easily work in your own instead

CHAR pszTemp[255];

        Order oOrder;

        Contract oContract;

        Contract::ComboLegListSPtr comboLegs;

        comboLegs.reset( new Contract::ComboLegList );

        comboLegs->reserve( 2 );

        if ( nLots == 0 )

                return -1;

        ComboLegSPtr leg1( new ComboLeg());

        ComboLegSPtr leg2( new ComboLeg());

        leg1->conId = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture1].oContract.conId;

        if ( g_Config.m_pSpreadInfos[nSpread].nContracts1 > 0 )

        {

                leg1->ratio = g_Config.m_pSpreadInfos[nSpread].nContracts1;

                leg1->action = "BUY";

        }

        else if ( g_Config.m_pSpreadInfos[nSpread].nContracts1 < 0 )

        {

                leg1->ratio = 0 – g_Config.m_pSpreadInfos[nSpread].nContracts1;

                leg1->action = "SELL";

        }

        leg1->exchange = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture1].oContract.exchange;

        leg1->openClose = 0;

        leg1->shortSaleSlot = 0;

        leg1->designatedLocation = "";

        leg2->conId = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture2].oContract.conId;

        if ( g_Config.m_pSpreadInfos[nSpread].nContracts2 > 0 )

        {

                leg2->ratio = g_Config.m_pSpreadInfos[nSpread].nContracts2;

                leg2->action = "SELL";

        }

        else if ( g_Config.m_pSpreadInfos[nSpread].nContracts2 < 0 )

        {

                leg2->ratio = 0 – g_Config.m_pSpreadInfos[nSpread].nContracts2;

                leg2->action = "BUY";

        }

        leg2->exchange = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture2].oContract.exchange;

        leg2->openClose = 0;

        leg2->shortSaleSlot = 0;

        leg2->designatedLocation = "";

        comboLegs->push_back( leg1 );

        comboLegs->push_back( leg2 );

        oContract.symbol = "USD";

        oContract.secType = "BAG";

        oContract.exchange = "SMART";

        oContract.currency = "USD";

        oContract.comboLegs = comboLegs;

        TagValueListSPtr smartComboRoutingParams;

        smartComboRoutingParams.reset( new TagValueList );

        smartComboRoutingParams->reserve(1);

        TagValueSPtr tagValue( new TagValue());

        tagValue->tag = "NonGuaranteed";

        tagValue->value = "1";

        smartComboRoutingParams->push_back( tagValue );

        oOrder.action = szAction;

        oOrder.totalQuantity = nLots;

        oOrder.orderType = "MKT";

        

        oOrder.smartComboRoutingParams =  smartComboRoutingParams;

        

                ORDER_ID = g_nOrderID++;

                g_pClient->placeOrder(ORDER_ID, oContract, oOrder);

[Q] Combo order example:

[A] View/Reply Online (#44252)

Added on 23-May-2020

This works fine here:

        contract = Contract()

        contract.symbol = "AAPL"

        contract.secType = "BAG"

        contract.currency = "USD"

        contract.exchange = "SMART"

 

        leg1 = ComboLeg()

        leg1.conId = 403716459

        leg1.ratio = 1

        leg1.action = "BUY"

        leg1.exchange = "SMART"

 

        leg2 = ComboLeg()

        leg2.conId = 379536992

        leg2.ratio = 1

        leg2.action = "SELL"

        leg2.exchange = "SMART"

 

        contract.comboLegs = [leg1, leg2]  

       

        order = Order()

        order.action = "BUY"

        order.orderType = "LMT"

        order.totalQuantity = 4

        order.lmtPrice = 0.01

       

        self.placeOrder(self.nextOrderId, contract, order)

I would check the API log.

Also "ratio" field MUST be integer, not float:

Yup, that is the tip.

I did see a mention of a similar story where the _actual_ issue was obscured because the NumberFormatException was happening when the Java code was trying to format a prior exception.

Useful info from the API log is shown below. It shows the NumberFormatException for "SMART", but also shows another (as a traditionally formatted Java exception & call stack) for the value "1.0".

So it turns out the real issue is that I had specified floats for the "ratio" field in each ComboLeg.

Looks like ints need to be supplied for those, because once I did, it works fine.

Thanks for the pointer, Josh.

Cheers,

Tim

[Q] Get info on all legs of placed combo order?

The use case is that I want to adjust the exit price for the order based on the percent of max gain/loss achieved and the days til expiration.  To do that, I need to get from the combo to the individual legs.

[A] by Josh in this thread

Added on 17-Oct-2016

(quote)

Partially answering my own question….

I'm now able to walk the ComboLegList in the Contract associated with the Order and retrieve the contract ID (conId) associated with each leg.  If I could retrieve the Contract based on the conId, I would be all set.  But I cannot find a get() or other method to retrieve the Contract.

Any ideas?

— Bob

——————————————————————————————-

Bob- You can call reqContractDetails with the conId/exchange (or even just the conId) to receive all the contract details.

Josh

——————————————————————————————-

Josh,

Thanks for the reqContractDetails tip.  For anyone else interested, here are a few more details on parsing combo orders.

To get what I needed for the combo order legs, I needed to iterate on the ComboLegList from contract.comboLegs in the openOrder callback to get the Ratio and Action for the leg trade, and then process the reqContractDetails call as suggested to get the Contract object for the leg, which in turn leads to the Symbol, Right, Strike, and LastTradeDateOrContractMonth..

— Bob

(end of quote)

[Q] Note: good-after-time (GAT) orders are not supported for ("generic") combo orders!

[A] by chuckfarley974@yahoo.com

Just a heads-up that took me a while to sort through this morning: Via API or not, good-after-time (GAT) orders are not supported for ("generic") combo orders, despite what the online documentation would have you believe.

[Q] Can place orders for each leg, but can’t place combo!? wtf?

[A1] by Kurt from this thread

If you can submit a request for the individual legs but can not submit a

request for the combo, then it may indicate that IB does not permit trading

on that combo. Others have noted that stock/stock combos are not available

for all pairs of stocks either. Testing, I see that future spread combos

are permitted for ES, but not for CAD december-september.

A quick call to IB will determine whether GLOBEX currency future combos are

supported.

-Kurt

[A2] by Kurt from this thread

To make a long story short, it looks like you did not specify the symbol

"CL" for the combo. I see the documentation is incorrect about this (saying

you should specify "USD"), and the API got more fussy about it a year or so

ago. Maybe that's your problem. And otherwise try "SMART" as the exchange

for the *combo* only since that is what worked for me.

Read on for the gorey details, which I already wrote before I figured that

out.

As a first step I would request market data for the combo, or historical

data. It is possible this might work even though placeOrder does not. If

so it might narrow it down to a specific fussiness about individual fields.

placeOrder has historically differed slightly from reqMarketData in the

Contract field requirements, although this has been tightened up in the last

year or so.

The only thing I wonder about is whether the exchange for the combo itself

should really be "NYMEX" or not. But I don't do futures. It makes sense in

the case of "SMART" option legs that the option combo be "SMART". But I

could also route an option combo to "ISE" even though I'm not sure "ISE"

would be correct for the legs. Just a thought.

I just tried entering this into my UI which handles some of the details:

CL:CLJ1@NYMEX-CL:CLH1@NYMEX

and I see that I can get market data (paper and real accounts) and also

placed an order in the paper account and got status PreSubmitted with no

errors in the log. However I see that my code filled in "SMART" as the

exchange for the combo even though NYMEX was used for the legs. And it

worked, so maybe try that, unless you really needed a directed order (which

costs you). The combo also worked regardless of whether I defined the far

or near leg as the long ("BUY") leg in the combo.

From my log, these are the Contract fields used for the legs:

14:42:47 sent MarketDataRequest 650 conId=37955043 symbol="CL" secType="FUT"

expiry="20110222" strike=0 right="" multiplier="1000"

exchange="NYMEX" primaryExchange="" currency="USD" localSymbol="CLH1"

includeExpired=0 secIdType="" secId="" comboLegsDescrip=""

14:42:47 sent MarketDataRequest 651 conId=37955046 symbol="CL" secType="FUT"

expiry="20110322" strike=0 right="" multiplier="1000"

exchange="NYMEX" primaryExchange="" currency="USD" localSymbol="CLJ1"

includeExpired=0 secIdType="" secId="" comboLegsDescrip=""

and for the BAG:

14:42:47 sent MarketDataRequest 652 conId=0 symbol="CL" secType="BAG"

expiry="" strike=0 right="" multiplier=""

exchange="SMART" primaryExchange="" currency="USD" localSymbol=""

includeExpired=0 secIdType="" secId="" comboLegsDescrip=""

And it looks like you did not specify the symbol "CL" for the combo. Maybe

that's your problem.

And otherwise if there is any needed info omitted from the above, let me

know and I can look at fields in the debugger, or add info to my logging

since my intent is to log everything that might be useful.

-Kurt

[Q] Is there a way to get contract details of the BAG contract itself?

Is there a way to get contractdetails of the BAG contract itself. I get the

following error when I try to request contractdetails.

MSG= Error validating request:-'dc' : cause – 'BAG' isn't supported for contract

data request. Please ent

er a valid security type ErrCode= 321

[A] by Kurt

There is no contractDetails for a BAG. This makes sense since a BAG is not

really a contract (even though the Contract structure is used to specify

it).

You may notice that execDetails for a BAG seems to have a Contract for a

bag, but if you look at the conId it is always 28812380, the same for all

BAGs. If you think about it it would be pretty hard to have a unique conId

for each possible N-way combo. I don't suppose even a 64-bit conId could

handle the general case, although in that case probably unique conId values

could be allocated dynamically by IB according to client use.

-Kurt

[Q] Combo example please!

[A] by A X <ax88@…> taken here

private Contract createBag(int conidOne, int conidTwo) {

           ComboLeg leg1 = new ComboLeg(); // for the first leg

           ComboLeg leg2 = new ComboLeg(); // for the second leg

           Vector addAllLegs = new Vector();

 

           log.info("conid in bag "+conidOne);

           leg1.m_conId = conidOne; 

           leg1.m_ratio = 1;         

           leg1.m_action = "BUY";   

           leg1.m_exchange = "NYMEX"; 

           leg1.m_openClose = 0;     

           leg1.m_shortSaleSlot = 0; 

           leg1.m_designatedLocation = "";

         

 

           leg2.m_conId = conidTwo; 

           leg2.m_ratio = 1;         

           leg2.m_action = "SELL";     

           leg2.m_exchange = "NYMEX"; 

           leg2.m_openClose = 0;     

           leg2.m_shortSaleSlot = 0; 

           leg2.m_designatedLocation = "";

 

 

           addAllLegs.add(leg1);         

           addAllLegs.add(leg2);

 

           Contract contract = new Contract();

           contract.m_symbol = "USD";     // For combo order use ?USD? as the symbol value all the time

           contract.m_secType = "BAG";   // BAG is the security type for COMBO order

           contract.m_exchange = "NYMEX";

           contract.m_currency = "USD";

           contract.m_comboLegs = addAllLegs; //including combo order in contract object

           contract.m_symbol = "CL";

 

           return contract;

       }

[Q] IB Lag with Combination Orders

[A2] by Kurt

On 7/8/10 7:54 PM, "coolwhs" <jjsmithdds@…> wrote:

> I occasionally use the combined order feature for vertical spreads,

> butterflies, etc. with options, and I've seen where the quotes change and hit

> my bid/ask which has been sitting on their exchange, and it takes a good 1-3

> seconds for IB to realize the price has been met and execute the order,

> occasionally missing the trade altogether. Does anyone have any similar

> experiences? Is there any way to get around this rather than just resubmitting

> the order and hoping the price is still the same? Thanks.

There is an option in the TWS presets for Combos which I'll bet is what is

burning you, but this is only for two-leg spreads. I'll quote from the TWS

manual:

Add Liquidity

If checked, this feature attempts to improve the price on the combo by

providing liquidity on at least one leg of a two-legged combo by submitting

one leg of a marketable combo one tick below the market for that leg.

Leaving this feature enabled may result in the combo order not executing.

For two-legged stock-stock combos.

I've found this can work really well, when it works. I've had a lot of

combo orders go through with very good prices with this option enabled,

which I think was checked by default, without my knowing of its existence.

However now and then I get burned and could not figure out why execution did

not happen when my offer was better than the best after some price movement.

So I called IB about the failed executions and they pointed me to that piece

of doc.

My impression from experience so far is that this feature works well for a

lot of things, including AAPL options, and it definitely does not work well

at all for options on SPDRs, etc. (maybe because they have a dense array of

strike prices?). I can't really draw solid conclusions yet but that is my

impression.

I don't know whether there is a way to express this option via the API, or

whether you just have to set it in TWS–never looked.

Regarding combos with 4 legs for example, I often try to decompose them into

two separate two-leg combos each of which has a relatively small net delta,

so that I do not mind submitting the orders separately and taking the risk

that they execute at somewhat different times. I've just had too many

executions fail with 4-leg combos, so I've just kind of given it up. You

might want to check whether the combo type you are interested in is native

to the exchange (maybe ISE). An IB rep told me once that combos beyond 2

legs may not be native. I never really checked whether this was true, but

just made my own adjustments to get more reliable execution.

I've also had particular executions problems with stock/option combos with

more than one option leg, and tend to decompose these as well.

Mind you the above is all based on a limited amount of *manual* trading

experience. People who do ATS stuff are more likely to have solid data

regarding these considerations. But OTOH it is not legal to do clickless

trading on US options.

-Kurt

[Q] Can’t change qty on combo order!

[A]

  • Eric J. Holtman
  • Message 1 of 7 , Jul 21, 2010
  • View Source
  • It's a bag to buy 10 strangles on a US Equities (so, one call, one put).
  • when I try, I get the following error:
  • error (5, 105, 'Order being modified does not match original order')
  • steps:
  • I place the order. It shows up correctly in TWS as the correct
  • combo, legs are correct, ratios, etc.
  • I later change *only* the totalQuantity field in the Order
  • struct, I do not touch the Contract struct at all, and I call
  • placeOrder again, and that error pops up.
  • So I know the error isn't of the type that can occur if I try
  • to use data returned back to me in a status message. It's
  • the exact same structures (except for that totalQuantity) field
  • each time.
  • Any clues?
  • Reply
  • Kurt Bigler
  • Jul 21, 2010
  • View Source
  • Do I recall someone saying that that for some kinds of orders modifying the
  • quantity is not permitted? If so this would be a case of a gibberish choice
  • of error message which we all know is typical of IB.
  • There was a thread maybe 9 to 12 months back about what fields can and can
  • not be modified in various scenarios. Maybe the native combo markets do not
  • like quantities to change or have unusual fees for such changes? I can't
  • imagine why though.
  • -Kurt
  • Show message history
  • Reply
  • Eric J. Holtman
  • Jul 21, 2010
  • View Source
  • On 7/21/2010 7:43 PM, Kurt Bigler wrote:
  • >
  • > Do I recall someone saying that that for some kinds of orders
  • > modifying the
  • > quantity is not permitted? If so this would be a case of a gibberish
  • > choice
  • > of error message which we all know is typical of IB.
  • >
  • >
  • Just tried changing either qty only or price only (I was doing
  • both at once). Neither one works.
  • It lets me do it inside TWS (of course, sigh).
  • [Non-text portions of this message have been removed]
  • Reply
  • Jason Sapp
  • Jul 21, 2010
  • View Source
  • Kurt,
  • Perhaps you are remembering the problems I had with modifying the quantity of
  • child orders of a parent order that was partially filled.
  • Jason
  • ________________________________
  • From: Eric J. Holtman <eric@…>
  • To: TWSAPI@yahoogroups.com
  • Sent: Wed, July 21, 2010 9:02:03 PM
  • Subject: Re: [TWS API] Modifying a bag order returns err 105
  • Show message history
  • Reply
  • Eric J. Holtman
  • Jul 26, 2010
  • View Source
  • After an Email from Ray @ IB, this has been solved.
  • I wasn't filling in the exchange field on the comboLegs.
  • Apparently, you can place orders without filling that in
  • but you can't modify them.
  • Once I changed it to put "SMART" in the legs,
  • everything works.
  • Show message history
  • Reply
  • Eric J. Holtman
  • Aug 13, 2010
  • View Source
  • On 7/26/2010 7:36 PM, Eric J. Holtman wrote:
  • >
  • > After an Email from Ray @ IB, this has been solved.
  • >
  • > I wasn't filling in the exchange field on the comboLegs.
  • >
  • > Apparently, you can place orders without filling that in
  • > but you can't modify them.
  • >
  • > Once I changed it to put "SMART" in the legs,
  • > everything works.
  • >
  • And of course, today, for no reason, it's stopped working
  • again.
  • I love TWS!
  • [Non-text portions of this message have been removed]
  • Reply
  • Eric J. Holtman
  • Message 7 of 7 , Aug 13, 2010
  • View Source
  • On 8/13/2010 1:35 PM, Eric J. Holtman wrote:
  • >
  • > And of course, today, for no reason, it's stopped working
  • > again.
  • >
  • Found it, replying here just in case someone searches the archives:
  • My problem was that I wasn't setting the "symbol" in the
  • main Contract (the BAG) to something sane…..
  • that is, I was always setting it to "GLD", even if I
  • was trying to trade a spread on "SPY" options.
  • It's weird that the API will allow me to place an order
  • like that (and it is correctly specifying options on SPY,
  • even though the main bag contract says "GLD"), but
  • it won't allow to me call placeOrder again to change
  • the price.
  • Once I changed it so that the main Contract has the
  • symbol SPY, everything works.
  • [Non-text portions of this message have been removed]
  • Reply

After trade is complete

[Q] How to get all Completed Trades within the last N days?

Is there a way to receive all completed trades via IB API? It seems to only return current day trades.

[A1] by Richard L. King View/Reply Online (#38974)

If you’re using the Gateway, you’re out of luck – there’s nothing you can do to return previous days’ trades.

 

For TWS, open the Trades Log window and you can specify how far back the information will go, up to a week. Whatever is set here is what you’ll get via the API.

 

Fortunately recent versions of TWS now persist this setting across sessions: in the not-so-old days you had to remember to set this every time you wanted more than just the current day, and you had to keep the window open or it was immediately reset. IBController used to automate this, but it no longer does because it can’t drive the current UI.

(update Jun 2018: Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.)

 

All of this underlines a point that is often made, namely that you don’t get anything via the API that isn’t available via TWS.

 

Richard

[A2] by Scott Kister yahoogroups@skister.com via groups.io View/Reply Online (#38975) 

You can set up flex queries which can be automated and fetched with a flex query token. Those queries are very flexible and could include asking for every trade and transaction for the current year.

–Scott

[Q] tell me more about Flex Query

[A1] by Jack Jost <jqjost@gmail.com> View/Reply Online (#39683)

Added on 03-Apr-2018

Historical EOD account data can be accessed via the IBKR FLEX API in XML format: https://www.interactivebrokers.com/en/software/am/am/reports/activityflexqueries.htm

Also:

If you only need execution info – just take a look at flex reports. As far as I know data can go as far as 5 years back (reports are also limited to 5 years)!

[A2] by Sal Abbasi abbasi.sal@gmail.com via groups.io View/Reply Online (#392)

I created a flex query that outputs data in CSV format.  If you go to account management / reports  / flex queries / settings you can get a token that will last a year.  The token allows your code to  run the query by using a URL for it including the report id and the token.  I then write the CSV data to a sqlite db.  These reports can give you all kinds of detailed data about past transactions, borrow costs, etc. that you cannot get from the API.

[A3] by Scott Kister yahoogroups@skister.com via groups.io View/Reply Online (#38975) 

You can set up flex queries which can be automated and fetched with a flex query token. Those queries are very flexible and could include asking for every trade and transaction for the current year.

–Scott

Also see official docs:

"Flex Web Service Version 3"

https://www.interactivebrokers.com/en/software/am/am/reports/flex_web_service_version_3.htm

Paper vs real accounts

[Q] Accuracy of paper-trading account

[A] read whole thread

— also —

If you want random data, use the Demo account, not Paper. It gives you

garbage nearly 24/7 that relates to the previous week's trading ranges, or

something similar. (Problem is most markets aren't open 24/7 and have an

open and a close, so which strategy were you going to test against this 24/7

trading US stock? It might help for finding some bugs. But Demo account

historical data was severely broken last I checked, and so your strategies

should not depend on any reference to historical data. It might work if you

start and stop your strategy at certain times since the Demo account goes

through some kind of cycles between which historical data gets reset (to

having no history).

The paper account has real data, identical to a live account, except for

market depth which has serious problems based on what people have reported

here. But it let's people test against real data when they need that.

If you don't need real data, it would be hard to create a worse simulation

than the Demo account. I'd think anyone who could would rather write their

own simulation, e.g. of markets that actually open and close and have high

volume at the beginning and end of the day.

(That's not quite what this thread was about, so you might start a new one

for the benefit of the archives if you want to continue the topic.)

-Kurt

[Q] PaperTrading – how do they simulate fills on limit orders?

[A] is funny and informative same time: here

[Q] Paper account vs. Demo account?

[A] by Kurt (from this thread)

If you want random data, use the Demo account, not Paper. It gives you

garbage nearly 24/7 that relates to the previous week's trading ranges, or

something similar. (Problem is most markets aren't open 24/7 and have an

open and a close, so which strategy were you going to test against this 24/7

trading US stock? It might help for finding some bugs. But Demo account

historical data was severely broken last I checked, and so your strategies

should not depend on any reference to historical data. It might work if you

start and stop your strategy at certain times since the Demo account goes

through some kind of cycles between which historical data gets reset (to

having no history).

The paper account has real data, identical to a live account, except for

market depth which has serious problems based on what people have reported

here. But it let's people test against real data when they need that.

If you don't need real data, it would be hard to create a worse simulation

than the Demo account. I'd think anyone who could would rather write their

own simulation, e.g. of markets that actually open and close and have high

volume at the beginning and end of the day.

(That's not quite what this thread was about, so you might start a new one

for the benefit of the archives if you want to continue the topic.)

-Kurt

[Q] Partial fillings of MKT order on paper trading account? wtf?

[A] by poch32 (found here)

Well, I think this could perhaps solve the "mistery" :

http://www.interactivebrokers.com/en/p.php?f=tws&p1=papertrader

from that link:

————

Fills are simulated from the top of the book; no deep book access

————

and just below that one:

—————–

"The trade simulator will reject the remainder of any exchange-directed market order that partially executes. This may or may not match behavior of a real-world exchange. Market orders received while there is no quote on the opposite side will be held until the market data arrives (i.e. until the first partial fill)."

—————–

So it seems that if you happen to enter a MKT order with a size bigger than the corresponding first DOM level, the order won't go thru the next level in order to get fully filled. And after that, the remaining unfilled contracts get rejected by the simulator… well, or something like that, I'd guess…

Seems a fair explanation, but… I cannot see how a tiny, miserable 6-contract order could overflow _any_ DOM level… anytime…

[A2] (quote from here)

Limitations of the PaperTrader

Although the paper trading account simulates most aspects of a production TWS account, you may encounter some differences due to its construction as a simulator with no execution or clearing abilities. These differences include but are not limited to:

  • No support for some order types including: VWAP, Auction, RFQ, and Pegged to Market.
  • Fills are simulated from the top of the book; no deep book access.
  • Limited combo and EFP trading.
  • Stops and other complex order types are always simulated in paper trading; this may result in slightly different behavior from a TWS production account.
  • Penny trading for US Options is not supported. You will be able to submit the order but it will not receive a penny fill.
  • The trade simulator will reject the remainder of any exchange-directed market order that partially executes. This may or may not match behavior of a real-world exchange. Market orders received while there is no quote on the opposite side will be held until the market data arrives (i.e. until the first partial fill).
  • Mutual Fund trading is not supported via IB PaperTrader.

(end of quote)

[Q] I am wondering if there is any way to tell from the API whether the socket is connected to a paper trading account or a real account?

[A] by Vlad Palnik

Paper account begin with a DU, execDetails.execution

[A2] by Castedo (added on 28-Apr-2015)

Dmitry's FAQ currently lists accounts beginning with "DU" as paper trading accounts, which is correct for non-advisor paper accounts. In addition, "FINANCIAL ADVISOR" paper trading accounts start with "DF". This is analogous to the "edemo" vs "fdemo" demo IB logins.

Both types, "DU" and "DF" paper accounts can be used for free with Brokertron Gateway for IB.

-Castedo (developer of Brokertron Gateway for IB)

[Q] Trying to place pegged to stock orders through API or TWS via paper account?

[A] by nirias999 (src)

If you are trying this against a paper account, it isn't working because the pegged orders aren't supported with paper accounts.


Charting

[Q] Any easy way to do charting?

[A] by Leno (found here)

Hello Vitali

I use Ninja Trader for my charting. It's free and the charts look great. Only problem, for me, is that the candlesticks times are at the end of the period, rather than the beginning, like with TWS and ESignal charts.

I use my own API for analysis and placing orders.

I haven't tried using Ninja for placing orders because I believe you have to subscribe in order to do that, but I'm not sure.

Good luck,

Leno

[A2] by Dmitry (27-Apr-2015)

Another option to draw timeseries using JavaScript in your browser is:

http://www.highcharts.com/

Here’s an example screenshot of 1sec bars view of historical data. When you zoom out highchart will rebuild bars size to show certain number of bars on the screen (you can see 5sec bar on the screenshot even though highcharts have 1sec bars data), which is configurable:

very similar result can be achieved by using Google Charts, but when you select time range with your mouse on the graph it would zoom into selected range instead of just scrolling timeline right-left. Also I’d rather go with highcharts since I found large community behind it and it was easier to overlap my datapoints (like orders or other custom indicator lines) on top of charts and it is generally working great for me (and I’m a bit lazy to study >1 framework for the same purpose:).

Apart from Highchaarts and Google Charts you can find lots of other charting libraries. Please visit collaborative comparison table about JavaScript Graph and Charts library for data visualization here:

http://socialcompare.com/en/comparison/javascript-graphs-and-charts-libraries


Fundamental data

[Q] reqFundamentalData() – How to Parse XML File?

[A] by souqmate from here

Little update on Reuters' Fundamental reports:

Since API 9.68 (I think), we can use new names for snapshot/finstat/estimates, viz ReportSnapshot/ReportsFinStatements/RESC.

We also have three new types, ReportsFinSummary, ReportRatios, CalendarReport, of which only the first has currently data.  In ReportsFinSummary we find 3 reportTypes (TTM/A/R/P – twelveTrailingMonths/audited/restated/preliminary?). Eg for MSFT: these 3 types differ for the EPS of 2013-06-30, but they don't differ for the Revenues.

The parsers from xml to csv for ReportSnapshot/ReportsFinStatements/RESC were offered on 20130708; here's the parser for the new ReportsFinSummary file (much easier!):

cat $x.ReportsFinSummary.xml | awk -F\" 'BEGIN{print "# what,curr,asofDate,reportType,period,amount (but 8 fields for Dividends: what,type,exDate,recordDate,payDate,declareDate,amount)"} ! /Summary/{if(/currency/) curr=$2; if(NF>3){split($0,a," "); sub("<","",a[1]); printf a[1]","curr","; for(k=1;k<=NF;k++) if(k%2==0) printf $k","  ; gsub(/<[^>]*>/,"",$0); sub(/  *|\.000000$/,"",$0); print $0}}' >! $x.ReportsFinSummary.csv

Eg:

DividendPerShare,USD,2014-09-30,TTM,12M,1.120000

EPS,USD,2014-09-30,TTM,12M,2.582009

Dividend,USD,CD,2014-11-18,2014-11-20,2014-12-11,2014-09-16,0.310000

# to print the Dividends (8 fields) into separate file: cat $x.ReportsFinSummary.csv | awk -F, 'NF==8' >! $x.ReportsFinSummary_divi.csv

Good luck,

souqMate.

[Q] How often Fundamental data updated?

[A] by Angel from IB support

Fundamental Data is updated directly by Thomson Reuters, IB delivers the raw XML data and does not manipulate/parse it for the API. Thomson Reuters updates the Fundamental Data daily. 

[Q] Why only 4 out of 6 fundamental requests succeed and for 2 of 6 I got errors:

– failed to fetch

– not allowed

[A] by Angel from IB support

"Falied to fetch" Meaning it is not being received by Thomson Reuters, more than likely they are not providing this report type. 

      And "not allowed" seems to be a bug (they're fixing it right now and will follow-up:) [as of Aug’2014]

[Q] You mentioned "we no longer directly provide support for API Ver. 9.67". How does this work? Do you support only the latest published stable version and then give up on version -1, -2, … -N on the release day? What are the rules here?

[A] by Angel from IB support

We publicly provide the current version, one previous build and (if available) a beta version. If there are any issues with the production version of the API Components we will roll out an update to that specific version to resolve said issue. For older builds you can still use it to connect to IB but we remove them from our public website and no longer provide support for fixing any issues.


IB Data Subscription

[Q] NYSE auction order imbalance data?

I just subscribed NYSE auction order imbalance data and found out the imbalance (field – 36) is total imbalance…

anybody knows how to receive market imbalance by field-36 ?

[A] View/Reply Online (#44333)

Added on 27-May-2018

It's available with the real time market data feed- you just specify generic tick type 225 in the generic tick list parameter of EClient.reqMktData, and then the auction data will be passed back during the opening and closing auctions in tick types 34, 35, and 36.

[Q] How to download broad market statistics, such as NYSE  TRIN, NYSE Advancing issues, NASDAQ ?

[A] by  Mario Pisa mario.pisa[at]gmail.com View/Reply Online (#40162)

Added on 04-Jul-2018

https://www.interactivebrokers.com/en/software/tws/usersguidebook/mosaic/marketStatistics.htm

https://www.interactivebrokers.com/en/software/tws/usersguidebook/technicalanalytics/tick___trin_market_indicators.htm

— also —

They're available from the API, e.g.

 

Symbol: TICK-NASD

Security Type: IND

Exchange: NASDAQ

Currency: USD

[Q] tickPrice vs tickString?

[A] read this interesting thread (about 38 messages). 21-Nov-2017 update: after groop moved to https://groups.io/g/twsapi the link to original discussion is now broken : (    todo: try to find the discussion and update the link…

[Q] Can I get historical price data of instruments which have expired?

[A] by J G View/Reply Online (#38987)

You can get historical price data of all instruments which have expired up to two years ago. I don't know how long price history currency options have. Let's assume six months. Then the oldest price you can get is if you take the option which expired 2 years ago and ask its entire price history. The oldest price point would then be 2.5 years ago.

Does this answer your question?

[Q] Trust daily close or 1-minute data close?

I am comparing the Daily (stocks) and the 1-minute data I pulled from IB, and there are quite a few differences.

Mostly it's the Close that differs, and on average by about 2%. This seems quite large to me. When I filter out the low-liquidity stocks, there are still differences, though a lot fewer.

From what I've read here, the difference may occur when a number of orders come right at the close and some go a few milliseconds past the closing time, which get bunched up to the next minute but will end up being counted as daily close.

Given the potential difference of 2%, what do you trust for backtesting?

View/Reply Online (#42498)

[A] by Julian Sommerfeldt julian.sommerfeldt [at ] gmail dot com View/Reply Online (#42516)

Added on 31-May-2019

There are some things to consider for the difference between those close prices: Auction/Time/Exchange

1) There is a close price coming from the auction at the primary exchange. This auction can take place depending on the exchange anytime after including 16:00:00, but also even some minutes later. I would say in most cases in the financial world this is the used close price.

2) There is a price coming from minute data and the last minute is 15:59 which ends at 15:59:59 not 16:00:00. So the normal close (and auction) price isnt even included in the last minute.

3) Be aware there can be a big difference if you request data directed to an exchange or as SMART (including all available exchanges).

You can get a good understanding by looking into the "Time & Sales" window showing each trade with its exchanges and conditions, e.g. auction and additional information why a trade is not reported as last price. Additional take a look at a minute chart and set "Show data outside of regular trading hours" to true, e.g. MSFT 2019-05-30 16:00 has a very much higher volume than 15:59 because it contains the auction and this minute bar is normally not shown and not included in the regular trading hours minute data.

So if you want to know the close price of the primary exchange you can do a directed request for dailys. But if you are routing SMART and not e.g. MOC orders (participating at the auction), also the data around the closing can be interesting. At the end it is a question of your trading approach.

Have fun,

Julian

[Q] VWAP through the API

[A] by MattScarpino mattscar@gmail.com

View/Reply Online (#42734) 

Added on 18-Jul-2019

If you're looking for real-time VWAP values, you can call reqRealTimeBars and set whatToShow to TRADES. You can also call reqMktData and add 233 to the genericTickList.

[Q] What RTVolume fields are?

[A] by rwk (from this thread)

When you specify generic ticktype "233" in the third parameter to reqMktDataEx() to request RTVolume the data is returned in a callback to tickString() in a single string containing six semicolon-separated data items like this:

1239.25;50;1312388185479;2439001;1244.86324288;false

According to Raymund in the API group, RTVolume is "realtime volume", and the six data items returned are last trade price, last trade size, last trade time (EDITOR UPDATE: this “last trade time” is actually generated on the TWS/Gateway side (client side) and is not passed by IB via network.. It is completely fake and bogus see this discussion for more details), total volume, vwap, and single trade flag. This is good stuff, and just what I have been looking for.

By way of background, the analysis of an order book, also know as depth of market (DOM) or Level II (though the latter refers specifically to Nasdaq) actually requires two data streams. The first is the order book itself, which contains only orders (quantity bid and offered at each price), and no trades. The trades show up on the [ticker] tape, also known as time & sales. Either by itself is not enough information to build a traders' workstation.

The problem I was having was that reqMktDataEx() returns last price and quantity in separate callbacks, and mating them was extraordinarily complex. It was in sorting through the various discussions here that I found Raoul's thread. It appears that the RTVolume was created to solve just this problem. Unfortunately, it is mostly undocumented, and the semicolon-separated string format is not exactly the most user-friendly. By specifying the "mdoff" flag on the tick generic parameter, all unnecessary market data is turned off to save bandwidth. The ticker tape is returned in tickString(), and the order book (bids and offers) is returned in updateMktDepth(). This kicks butt.

Now we have "documentation". πŸ™‚

[rwk]

—————- and also —-

Btw. the method described above is applied on api version 963, so I'm not

sure if newer versions have changed anything since then. Upgrading means I

have to retest every little callback like I did at the time.

Greetings,

Raoul

[Q] Realtime Volume timestamp sometimes goes backwards?

[A] from this thread

Added on 13-Sep-2016

Yes, it does sometimes go backwards because it does not exist. πŸ™‚

"Realtime Volume timestamp" is not transferred by IB, but instead is generated locally by TWS/Gateway application, which in turn is adjusting it's local time knowledge about every 3-7 minutes. Local TWS/Gateway time is adjusted instantaneously with up to 1.4sec value (usually less: 100-200ms). If IB/Gateway time jumps backwards 1.0s right in between 2 RTVolume messages then 2nd will seems to be in a wrong order even though it is not.

pps: Or may be those ticks are actually sometimes delivered in a wrong order, but there's no way to tell having only IB's data subscription as IB decided not to share their time knowledge.

[Q] Is there an easy way to get the S&P 500 index component stocks?

[A] by souqmate from this thread

Added on 17-Feb-2016

there’s no easy way to do this by using API, but …

Note that you can still do it manually with the TWS:

TradingTools -> BasketTrader -> IndexPanel -> choose SPX@CBOE as Index

set Amount=10000 shares, click on Create Orders,

and 504 rows of order will appear  (don't click on Transmit !).

You can export these via File -> SaveBasket

The csv file wll contain tickers.

I doubt the API can do the same.

Gd luck

S

Editor addon: also to parse the file in bash:

resulting file has following structure:

(quote)

Action,Quantity,Symbol,SecType,Exchange,Currency,TimeInForce,OrderType,BasketTag,OrderRef,OrderCreationTime,

BUY,100,A,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,AA,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,AAL,STK,SMART/ISE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,AAP,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,200,AAPL,STK,SMART/ISE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,ABBV,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:11 EST,

BUY,100,ABC,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,ABT,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,ACN,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:11 EST,

(end of quote)

# now let us parse it and extract 3rd field and skip 1st header line:

cut -d , -f 3 snp500_basket.csv | tail -n +2

A

AA

AAL

AAP

AAPL

ABBV

ABC

ABT

ACN

ADBE

ADI

* * *

[Q] Can I download the historical data of obsolete symbols?

[A] by Vishruth Vasistha from this thread

Try setting IContract's IncludeExpired to 1. You should be able to get

historical FUT data.

Thanks,

Vishruth

[Q] My accumulated volume does not match the TWS one! Wtf with VOLUME?

[A]  read this discussion

[Q] Does it make sense to use separate data / broker connections for separate tasks?

Does it make sense to connect to TWS

– for data related services with a client ID, and

– for broker services with another client ID

from the same application? Is there any pro/con using this approach?

Thanks,

Zsolt

[A] by Richard L King

My opinion is that in general it would probably be of no real benefit, but

it would depend heavily on your application.

I guess you're thinking that you'd get order-related events more quickly

than if they were mixed up with the market data stream, and this is true but

the advantage may be negligible. Assuming that it doesn't take you more than

a very small number of milliseconds to process tick events, most of the time

you'd probably get no advantage at all because there are no outstanding

ticks to process. But of course if you're handling a large number of busy

market data streams, and perhaps one or more market depth streams, then this

might alter the picture considerably. I suppose it all depends on how

critical the timely receipt of order-related events is to your application.

If a delay of say 100 milliseconds (over and above the natural latency of

the exchange to IB to TWS to client connections) would be important, then it

might be worthwhile.

Because it's bound to add some complexity to your application, my advice, as

for any optimisation, would be not to bother until you've developed and

tested the application and then only do it if there is a demonstrable need.

There are a couple of other points worth adding here:

– TWS maintains separate connections to market data, historical data and

account servers (and my assumption is that order-related events go over the

account server connection, though I've never verified this by examining the

traffic). So it's only the API connection where order-related events might

get held up;

– when you make a historical data request, the whole of the data is returned

in a single API message. If your app makes large historical data requests

and/or processing of historical data is relatively time-consuming (updating

a chart perhaps), then it might conceivably be worth making these with a

separate client id.

Richard

 

[Q] API and Trading hours?

[A] by Richard L King (src)

The contract details contains trading hours information for the current and the next day’s session. I don’t use it myself, so I can’t vouch for its accuracy, but whenever I’ve looked at it, it seems pretty good.

I’m mildly surprised that you could have got as far as producing an auto-trading system without having noticed this information: it’s been in the contract details for years. Not a criticism, just an observation.

Richard

[Q] How to get the day open price for NYSE stocks?

[A] by Josh in this thread

Added on 12-Oct-2016

Sorry Nasdaq stocks are actually on Network C. Since AAPL trades natively on Nasdaq, you would need the Network C subscription, since only the Nasdaq exchange has the official opening price.

Network A would give you the official open for stocks on Nyse, and Network B would give you the official open for stocks on Arca.

Without a subscription to the native exchange, you will just receive the first traded price on a different exchange.

[Q] How to get Futures Open Interest?

[A1] by mattpearson911

Added on 24-Nov-2016

You can get daily CME/CBOT/COMEX/NYMEX open interest and other related info from the CME ftp site:

ftp://ftp.cmegroup.com/settle/

For example: ftp://ftp.cmegroup.com/settle/stlcomex

[A2] by btw12342001

Added on 24-Nov-2016

In addition to CME directly, you can get the data from Quandl which has some APIs for retrieving data.

CME NYMEX WTI Crude Oil Futures (CL) – Data from Quandl

[A3] by Isaac

Added on 24-Nov-2016

Or, try TickData.com.

Update from OP: Tickdata.com is very expensive given that I am only interested in the open-interest information (which is only available on a daily basis anyway), but CME ftp server and Quandl are excellent tips.


Data subscription: Volume

[Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] What is the sequence of tickSize() tickPrice events?

see also relevant: [Q] Building live Trades

[A] by Mike Smith (from this thread)

if you monitor the cumulative volume AND keep your own total of cumulative volume which is the sum of the "size" parameter of each "last price" sample then you have a cross-check. When your total of cumulative volume is less than IB's reported cumulative volume then samples were missed. My approach is to add that difference at the current "last price" as if a trade was made of that size. That brings my cumulative volume in sync with IB's reported cumulative volume. Doing this each time there is a cumulative volume mismatch gives me fewer trades of larger size compared to the many more trades of smaller size which actually came from the exchange. The result is all the volume is accounted for, Market Profile tables look pretty close and life is sweet – except for one thing. The API's cumulative volume does match TWS Chart's cumulative volume.

and then by btw12342001

I do the same thing except I just remember the previous cumulative volume and subtract it from the current value. It works out to the same thing except if for some reason you don't get all the Cum vol (tickType 8) messages. This happened to me in Nov. I've since re-done my code to use the tickSize (5) message if I'm not receiving 8's

and back to MikeSmith

I have spent a lot of time with both screens in front of me and I have never seen TWS update last price without a corresponding reqMarketData update. The only way I could see this happening is if TWS is streaming level 2 quotes for that symbol which means it gets more frequent updates. Might this be the case?

As for size events, I don't look for size events less than 10 ms apart or depend on timing in at all. I look for a last price event followed by a size event with no other events in between. Then if another size event arrives which is the same size and again, with no other events in between, then I ignore that size event. Then I ALWAYS look at the difference between IB's reported cumulative volume any my running total and if my total is smaller then I create a "last price/size" event out of thin air using the difference for the size value. This takes care of those rare times when I should NOT have ignored the size event, and fills in missing samples to keep the total volume in sync as well. I found this more reliable than depending on timing.

Mike

btw, in that thread Richard L King explains why IB come up with 2msg in 1 message (tickSize and tickPrice). I will copy-and-paste it here, so you don’t miss it:

[Q] How to get volume from all exchanges vs. volume from one exchange only?

[A] mentioned here by Josh: View/Reply Online (#42746)

When requesting market data for stocks, its most common to use exchange = "SMART" so you can receive the NBBO and aggregate volume from all exchanges. So you can create a Contract object with the returned conId, and then just set the exchange = "SMART", and use that to request market data.

(If you wanted to request data from Nasdaq only, then you would have to have set exchange = "ISLAND" rather than "NASDAQ" in your request).


Data subscription: Dividents

[Q] Get Dividend information?

[A] by Kurt

I can't speak to the API but tend to believe the API never functions

*better* than TWS. And TWS has proved to be pretty bad re dividends.

For a while I was playing around with EFPs, using IB's standard scanner

settings "Relevant EFPs" under the Excess Cash category. Well one day the

scanner popped up an EFP that looked really attractive to write. It turned

out that the dividend had just been announced (as nonexistent, I think or

else much lower than anticipated), but that IB's dividend info was still

showing the originally predicted dividend. This mean that the EFP's yield

valuation was way out of line with reality, looking really good, because the

future was already priced based on the knowledge that the dividend was

lower, or gone, whereas TWS was not only displaying the wrong dividend, but

computing the EFP yield incorrectly. So writing the EFP was an instant

loss, but I didn't find out until the next day.

And I believe it happened to me more than once, although it took me a while

to figure out what was going on.

In short as IB says, dividend information is provided … blah blah blah.

In other words, they have so little accountability that they are free to

screw up in major and obvious ways with no consequence to them.

So although it doesn't answer your specific question, I wouldn't trust IB

dividend info for *anything*.

(I'm guessing the API dividend info would match what shows in TWS but I

haven't checked it.)

[Q] Dividend Calendar (dec, rec, div, exdiv, amount, etc)

[A] by Josh View/Reply Online (#39066)

Added on 08-Dec-2017

The only other place for dividend information is the Financial Summary Reuters report (API function reqFundamentals). It gives historical dividend information, including for dividends which have been declared but not yet distributed.

[Q] Dividend Adjustments? Adjusted Previous Close not available on IB?

I'm trying to check for upcoming dividend payments to ensure that all indicators are correctly adjusted and to ensure that the price drop does not create a false trigger.  Although I've read posts saying that IB adjusts for dividends, this does not appear to be the case with this example so I'd love to hear from people with experience.

For example, historical IB price data for SPY (S&P 500 ETF) appears to be the raw data.  The ex-div date for SPY was 20-Dec-13 and the dividend amount was $0.98025.  

Data from IB via req_historical_data for Trades:

Date

Open

High

Low

Close

Volume

Dec 20, 2013

180.67

181.99

180.66

181.56

1,131,358

Dec 19, 2013

181.19

181.70

180.71

181.51

927,670

Data from finance.google.com:

Date

Open

High

Low

Close

Volume

Dec 20, 2013

180.68

181.99

180.57

181.56

197,086,908

Dec 19, 2013

181.19

181.70

180.71

181.49

136,298,872

Data from NASDAQ:

Date

Open

High

Low

Close

Volume

Dec 20, 2013

180.67

181.99

180.5683

181.56

196,525,900

Dec 19, 2013

181.19

181.70

180.71

181.49

136,558,000

Data from finance.yahoo.com:

Date

Open

High

Low

Close

Volume

Adj Close*

Dec 20, 2013

180.69

181.99

180.57

181.56

197,087,000

181.56

Dec 20, 2013

0.98 Dividend

Dec 19, 2013

181.18

181.70

180.71

181.49

136,531,200

180.51

It appears that only yahoo adjusts the closing price for the dividend payment (also interesting to note the volume discrepancy with IB compared with the other sources).

Does anyone know how or if IB adjusts historical prices for dividend payments and how to anticipate them?

Thank you.

 -BB

[A1] by souqmate https://groups.io/g/twsapi/topic/4046961

warning: IB adjusts stock prices for splits and stock dividends, but not for cash dividends nor mergers nor spin-offs, etc.

Perhaps subscribing to Reuters StreetEvents Calendars ($10) will provide you with future dates and amounts of cash dividends.

IB generates sometimes wrong jumps in its price series of stocks, as some (reverse) splits are made to offset spin-offs, as happened to  UNTD on 20131101; IB and google claim factor 1:7 (huge jump), while yahoo has a more realistic factor of 5.319 : 7 (due to simultaneous spin-off of FTD).

You can vote here to have corporate actions calendars accessible via the API:

https://www.interactivebrokers.com/en/index.php?f=2493&sid=9407   (fixed link)(still works Jun-2017:)

Aside, your volume discrepancy is due to an implicit factor 100 in IB's volumes on US stocks, and perhaps to your restriction to a certain exchange (instead of SMART) or to certain times;  SMART gives me a volume of 1,466,263 on Dec 20.

souqMate.

You can vote here to have corporate actions calendars accessible via the API:

[A2] by souqmate@yahoo.com

pls search the archive before posting.   See eg

https://groups.io/g/twsapi/topic/4046961

Beware that historical data may not be availabe for several days around splits dates, as it happened for BP@BVME (1:10 split on 20140310, had no data for 3 days) or for IMI@LSE (7:8 split on 20140217, had no data for 2 weeks).

Further, some factors are inserted arbitrarily though there are no splits, eg SWN@NYSE prior 20130102 has prices 100* bigger (used not to be the case, but then such things suddenly appear…).

And as a side note (no splits here), beware that some gaps appear even in liquid symbols like Canadian futures, which have no data on 20140312 (TSE60, CGB, BAX) and that IB has lost the data for good.

Thank IB to help you see glasses half-full rather than half-empty.

souqMate.

[Q] API reqFundamentals dose not return previous dividend data for index

[A] by Josh and couple others View/Reply Online (#42651)

Added on 24-Jul-2019

Unfortunately no theres not really anything similar available for index data from the TWS API.

As a workaround:

For indexes, you need to choose a proxy, such as SPY for SPX.

Or as ETF dividends which are not really any different from equity dividends from a data perspective.

[Q] Data subscription: Calendar events

Hi all,

If you go into the Event Calendar option in TWS you can see various upcoming events, e.g. we are interested in Corporate Earnings.

There do seem to be some API's that relate to calendar events but it seems you need a subscription to Wall street horizon to download the data and the page to subscribe doesn't even seem to be accessible – maybe it's only available to professional accounts?

Is this the case or has anyone found a way to access this event data programmatically? I'm thinking of looking into some sort of windows automation tool to see if I can "scrape" the data from TWS.

[A] by alpha <pmularien@gmail.com> View/Reply Online (#44591)

Added on 17-Jun-2020

Yes, you can get this data with a Wall Street Horizon subscript (available under User Settings > Research Subscriptions). I am a non-professional and I have a subscription ($5/mo).

If you're using the Python (IB-insync) API, the code to get this data looks as follows:

ib.reqFundamentalData(contract, 'CalendarReport')

This will return XML data which is pretty straightforward to parse.


[Q] I got invalid update of bid/ask which leads to CROSSED PRICE. wtf?

[A] by Kurt (from this thread)

> I noticed that IB sends updates for bid/ask in invalid order sometimes. E.g.

> bid/ask was 45.47/45.48. Then IB sends new bid 45.99 which leads to crossed

> price 45.49/45.48. The next moment IB updates ask to 45.50 and the price is

> back to normal – 45.99/45.50.

>

> How do you guys handle that rubbish price of bid 45.49 / ask 45.48?

> Thanks

I don't know that there is any general answer. There is constant potential

for invalid/stale quotes. I don't think you are getting bid and ask

literally in the "wrong order". Rather one of the two is wrong. Was the

size non-zero on both sides? You didn't mention what instrument.

I have seen maybe not crossed bid/ask, but very narrow or equal bid ask, and

basically the bid or the ask was not real, and the question is which one. I

would find out the hard way when an order against one would not execute. I

believe that in every case I saw the problem was specific to a single

exchange which had the bad quote contributing to the "SMART" quote. Once I

recognized the problem and intervened to enter quotes lines (manually, into

TWS, but you could do it automatically with reqMktData rather nicely), I

generally found that the primary exchange did not have the problem and that

some particular exchange (e.g. maybe BOX, maybe once ISE or was it ISLAND)

had the bad bid or ask.

This problem has been recognized here and alternatives discussed about how

to deal with it most effectively. People have discussed finding the

exchange that has the most volume, etc., but I think the primary exchange is

good as a general rule. Then you can get quotes from a less-potentially

polluted source than SMART. SMART can be a real liability at times as

people have commented. Basically if any exchange at all has a bad quote,

SMART will be bad, so by using SMART for quotes you maximize the potential

for problems. So if you care about this you may want to request quotes from

a specific exchange but still place your orders on SMART to get the

commission and execution advantages, depending on all the details of order

type etc.

You might search the archives for previous comments on this topic, which I

think might be 6 months or more ago. Doing a quick search in my email

folders, I see a thread "bid/ask stuck" with posts on 6/22/2009 and

6/23/2009. But I think there is something more recent and with additional

observations. If you can't find anything (yahoo archives can be a

PITA–better to have your own repository) post back and I'll take another

look. I know that I was involved in most discussions of this in the last 2

or 3 years, which might give another clue for a search.

-Kurt

[Q] Crossed quotes on IB (again)

Added on 27-Apr-2015

I noticed the market cross on TWS where the quote on my screen showed up as 11.45 x 11.44 for a few seconds…

[A] by Mike Smith (found here)

What did you specify for the exchange? If you used SMART you will get crossed markets. For BAC you should use NYSE for the exchange. SMART is designed for picking the best exchange to route orders and for market data it is not as good as specifying the exchange with the highest volume for that stock. At least that's what I have found in practice, the "why" of it is another matter.

Mike

———- also ——————

You can see what is happening if you just fill the TWS window with BAC, but choose a different exchange for each line. Also include SMART on one line. If you watch all the bid/ask prices you can see which exchange SMART got it's data from. Exactly what it is doing is unclear but grabbing prices from different exchanges like that definitely can yield weird results. One exchange may show the same bid/ask price for many minutes, on another exchange it is bouncing around like jumping bean, yet SMART will occasionally give you the very stale data from the "dead" exchange which is very much NOT a tradable price.

Mike

————— and also ————–

It means one of the exchanges is crossed. SMART includes data from all of

them. If one exchange is crossed then the "best" will be crossed.

Like Mike said create market data lines (or API requests) for each exchange

and you will see which one is crossed and is causing the problem. In most

cases the bad data comes from one of the minor exchanges or at least not

from the primary exchange.

-Kurt

[Q] Snapshot market data vs “real time” data.

I've seen people say that snapshot market data is less accurate, reliable, etc. I'd like to learn a bit more about the specifics of why it's considered inferior.

Where have you observed snapshots providing incorrect or less data whereas the streaming service was fine?

Does the snapshot sometimes not provide all fields (bid, ask, open, etc)?

Is there any discussion regarding the latency of the snapshot vs. the real time data? For example, if a ticker price gets updated to $19.11 from $19.10, how far behind the real time stream will a snapshot request be?

[A] by Rich Holowczak (a.k.a. “Prof. H.”)

Added on 16-Aug-2016

Using reqMktData, the difference between snapshot and streaming is that once a value is provided for each field (bid Price, AskPrice, bid size, ask size, last price, volume, etc.) the snapshot is done and the request is effectively canceled. I pointed out the caveat to this before that if a field has not updated in the prior 11 seconds, it will not be echoed back with the snapshot:

http://interactivebrokers.github.io/tws-api/top_data.html#md_snapshot

If you need every field, or you need to make 100% sure you are getting a value echoed back, then you should request streaming data until your fields get a value, and then cancel the request yourself.

An alternative to this is to get real time bars which would get you a set of data every 5 seconds.

Using reqMktData, the difference between snapshot and streaming is that once a value is provided for each field (bid Price, AskPrice, bid size, ask size, last price, volume, etc.) the snapshot is done and the request is effectively canceled. I pointed out the caveat to this before that if a field has not updated in the prior 11 seconds, it will not be echoed back with the snapshot:

http://interactivebrokers.github.io/tws-api/top_data.html#md_snapshot

If you need every field, or you need to make 100% sure you are getting a value echoed back, then you should request streaming data until your fields get a value, and then cancel the request yourself.

An alternative to this is to get real time bars which would get you a set of data every 5 seconds.

[Q] Market Data Stuck when requesting snapshots!

particular question was:

Hi,

I have been having trouble from time to time where the market data seems to get

"stuck" on certain values, and does not change them even though they really do

change.

This happened to me today for over an hour with the "ES".

On one system the value got stuck, meaning that I request market data every

minute (snapshot), and received the same response each minute.

On a second computer with a different account, same code, it did not happen at

this time, but also happened later on.

Also, can anyone explain the order of the responses I'm receiving.

I see Price, Price, Price, Size, Price, Price, Price, Generic, Generic.

Thanx,

Rebecca

[QUICK SUMMARY ANSWER] Don’t use snapshots.

[A1] see discussion:

https://groups.io/g/twsapi/topic/4047462#32976

[A2] by Kurt

I never used snapshots much. I played around with them once and saw really

strange things. When I reported them rwk replied:

Hide message history

On 9/17/10 6:12 AM, "rwk2095" <rwk2095@…> wrote:

> I can't help with the snapshot problem because snapshots never seemed to work

> for me.

And if you want to see the details of the problems I had messing with

snapshots, see my post

"[TWS API] No "Last" for AAPL today?" on 9/16/10

Maybe they just don't work!

===

However others seem to be using them and not having problems. Maybe

snapshots don't work well with certain instruments?

I would call in to IB about this. Really you should have called yourself

before it was too late. As has also been repeated a few times recently

there have been problems reported as being due to particular servers in a

pool. This is *somewhat* consistent with what you report although it may be

a different twist. But if an immediate problem is associated with a

particular server in the "farm" that you are connected to, you need to call

IB during market hours and before you lose your connection, else no

diagnosing can be done.

> Also, can anyone explain the order of the responses I'm receiving.

> I see Price, Price, Price, Size, Price, Price, Price, Generic, Generic.

Without the tick# values this doesn't say much, so you might as well re-post

so that people can answer without having to second-guess.

And just to be clear (which I wasn't) by tick# I meant BID, ASK, LAST, etc.

Price/Size/Generic is secondary information, and redundant once you have

specified BID/ASK/LAST etc. So if you report it here that way you may get

some information back. There has been some discussion before about the

order that ticks arrive and what it means. Some of it is determined by the

so called "sample code" (really the library), which you can read yourself,

i.e. the place from which tickPrice and tickSize are called is from source

code provided by IB which is part of your code base.

-Kurt

[Q] reqRealTimeBars vs reqMarketData – how to get real time bars?

[A] by Kurt

reqHistoricalData lets you have 3-minute bars, but only gives you backfill.

You need reqRealTimeBars. However, there is no 3-minute option. Bars come

every 5 seconds once you subscribe. You will have to coalesce them the

5-second bars into 3-minute bars, but that should be easy enough.

However I know people have posted here about having problems with real-time

bars. I can't remember the details so you might check the archives. Part

of it is that people expect the bars to match the streaming data, which it

won't. But I seem to recall other problems, like maybe volume being 0 when

it should not be.

If you have trouble with real-time bars another option is to build your own

bars from tick data (reqMarketData).

Yet another option is to call reqHistoricalData on a timer, as you were

suggesting. However your latency will probably be worse because you have to

wait until the last bar is clearly complete before you make the request.

Otherwise the final bar would be truncated. (You should probably have logic

to detect the truncated bar based on the end time on the "finished" record,

and re-issue the request if needed, and refine your timing logic to minimize

latency without getting truncation.) You *can* use reqHistoricalData to

request a single bar (in cases I have tested).

-Kurt

[Q] How to get list of all IB’s tradable contracts?

[A] by Shane Cusson

I had some free time this weekend, I fired up a crawler. It pulls down

216,000 contracts from 224 exchanges listed on the products page.

I called it "IBContractExtractor", released under GPL. It's available here:

http://sourceforge.net/projects/ibcontractextra/

Shane has moved it over to GitHub at https://github.com/shanecastle/IbContractExtractor

It's written in C#, uses the HtmlAgilityPack. The pack is an awesome

library, takes all the grudge work out of parsing HTML, gives you a full OO

interface to the contents of the Html. Right now it's just pull out the

contracts, I'll make it extract all the other details at some point in the

future. The resulting text file is about 12MB, zips down to 3MB.

I don't know if it'll run on Mac/Linux yet, I did use some .net 4.0

features. You'll need .net 4.0 for it to run.

Please kick the tires, let me know how it works out for you.

–Shane Cusson

[Q] Sometimes lastSize=0. wtf?

[A] by mikesmithv (source thread: here)

Couriosity got the better of me so I turned on debug logging and looked at my code. It turns out that what I said is correct. You will loose data if you ignore lastSize events but it's complicated. I had to refresh my memory going over code I wrote some time ago but it appears not much has changed so here are the details for what it's worth.

First you have to multiply every size by 100 for stocks for both size and cumulative volume events. It is critical that you hold the expanded numbers in your code for what I say here to work.

Next, for size events greater than 1 (or 100 multiplied) you IGNORE them if they occur immediately following a size event of the same size AND that size event was proceeded by a price event. This is the infamous "duplicate size bug" which has been there for years. If you do not ignore this your running total of size will creep ahead of cumulative volume reported by IB.

Finally, for a strange twist, if the last size is zero then you DON'T ignore it. These zero size events represent trades of less than 100 as I mentioned earlier. Since everything is divided by 100 they get rounded to zero before they are sent out. You can have several trades at different prices with zero size, so that means price was fluctuating but at a volume that was under the radar of cumulative volume so just looking at cumulative volume delta will miss this. For example if you have trades of price $100, size 10, price $101, size 20, price $102, size 30 then you would see 3 price events of zero size and cumative volume would not have changed since 10+20+30 is less than 100 which is what it would take to bump up cumulative volume by one. That's IB's cumulative volume I'm referring to, yours is times 100.

So if you see zero size for stocks (never happens in Futures BTW) then SOME amount of volume occurred at that price. The question is how much? We just know it is something less that 100. I found 20 to be a good average. It worked well for matching Market Profile totals, as least as well as much more complicated schemes I came up with. You can be as creative as you want in coming up with this figure but it is important to post SOME quantity at the last price when the size is zero. It should be at least 1 so you have some indication a stock has traded through a price at sub 100 volume.

[Q] When is exact start of a new bar? Will hand-made bars (from tick data) be exact same bars as I receive from IB?

[A] by Kurt (src)

My memory about IB's bars is that we know that the start of a new bar

happens at a time that we, on the other end of the sampled data feed, can

not expect to identify. In fact the moment a new bar starts may not even

*exist* as a tick in the sampled data feed. So what you are getting is

basically the equivalent of "aliasing". If the price jumps up *before* the

point you think is a new bar, but IB's bar timing thinks that moment was

*after* the new bar started, then you will see the right data in the wrong

time slot. It might have been only 1/10 of a second difference in timing,

but sampling at the wrong point can make it look like it is a whole second

off. So you can expect this time skewing, and that is what your last plot

seems to show.

— also —

I don't have any information to the effect that IB's bars are based on

sampled data.

Regarding the "tick data" I've heard here that two separate connections get

different sampling. So it would seem (if it weren't already obvious) that

IB's systems have access to unsampled data. I would assume they would use

such unsampled data to generate their historical bars.

The point is that even if the historical bars come from unsampled data, they

can not be expected to match sampled "tick data" in the sense that the bars

could be reproduced from the sampled tick data.

–Kurt

— also —

It must be the case that IB take their input from unsampled data streams, surely.

Also, IB told me that it's sampled every 300ms – at least for forex, they

weren't explicit.

I assume they sample it to reduce down the ticks to 3 per second instead of the

random amount on the live feed.

What that tells me is that (a) what you mentioned about the real O and C for any

bar in the resulting dataset being fictional and (b) the H and L could be lower

or higher respectively than the actual H and L. (and of course (c) the H and the

L could appear in either bar on whichever side of a particular time point).

–Adam Hardy

[Q] Why IB uses sampled data instead of plain real-time tick streams? Aka "fudge" factor for tickSize event.

[A] by Richard L King

see:

https://groups.io/g/twsapi/topic/34510650

https://groups.io/g/twsapi/message/8821

Actually IB never accumulate anything (except total volume) and they don't

throttle. They just don't report every trade or every bid and ask.

You may find the following of interest. It is a composite of several posts

I've made here in recent months, and is based on some very intensive

research into how the IB data feed works that I undertook some years ago (as

far as I can see, it still works the same way today).

The IB datafeed is optimised to ensure that it keeps up with the market no

matter how busy the market is. As a result of this IB doesn't report every

trade, or every bid or ask, for any instrument in any exchange.

To accomplish this, it effectively sends a price snapshot for each

instrument at regular intervals. This interval seems to be about 300

milliseconds. For each of bid, ask, and last it compares the current price

and size with the values at the last snapshot. If the price is different it

sends both price and size. If the price is the same, but the size is

different it sends only the size. If both price and size are the same, it

doesn't send either. If there have been any trades since the last snapshot,

it sends the (accumulated) volume (so where the price and size haven't

changed but there have been one or more trades, this can be detected from

the increased volume).

By doing this, IB knows the maximum bandwidth required for each ticker, and

hence for each customer (since the number of tickers is limited), and so it

can size its servers to be able to cope with that load. If a market becomes

very busy, it makes no difference because it will still only send an update

three times a second or thereabouts, even if there have been 100 trades

during that second. This avoids the problem that every other data feed seems

to have, where the data will sometimes lag way behind the market at busy

times (with every other vendor I've used, I've had occasions where the data

could be anything up to two or three minutes behind the market).

There is an irritating side effect of this technique, which is that price

movements between shapshots may not be reported at all: for example if the

last price at snapshot 1 is 100, and then price moves up to 102 and then

back to 101 by snapshot 2, the price reported at snapshot 2 will be 101, and

the 102 price will not be reported at all. This leads to occasional

incorrect highs and lows of bars, but rarely by more than one tick: whether

that is significant depends very much on the trading strategy used.

In my experience, the number of occasions where the inaccuracies have any

significant effect is so small that it can be ignored, but that of course is

partly a function of the particular strategy I was using – note however that

it traded aggresively off 1-minute bars and used very tight stops, so you

might think it would be quite sensitive to inaccuracies. Besides, sometimes

the inaccuracies work against you and sometimes they help you, so overall I

think it pretty much balances out.

The above isn't a complete description, but it covers the basic mechanism. A

word of caution though: this is not an exact science. It would be nice if

what I said was an exact description of how it works, but you'll find odd

things happening occasionally, such as a volume update without a prior size

message where the increase in volume is not an exact multiple of the most

recent size message, or multiple last price/size messages sent at the same

time, or volume messages with a smaller volume than the previous one! But

most of the time my description is accurate.

Another point worth making is that the snapshots are taken at different

times for different TWS sessions. This is sensible from IB's point of view

because it means they don't have to send updates to all their thousands of

users at once. But it has the interesting side effect that if you run two

copies of TWS (eg different logins to the same account) you'll get slightly

different data from them, because the snapshotting is not being done

simultaneously.

By the way, one gotcha that I didn't mention above is that when both price

and size messages are sent (in a single TICK_PRICE socket message), TWS also

sends the size again in a separate TICK_SIZE message, but the volume is

correctly updated only once. I think the reason for this duplication is that

before the version 2 TICK_PRICE message was introduced, it didn't contain a

size field, so prices and sizes were always sent separately: if TWS didn't

send the duplicate size, then programs that relied on the separate TICK_SIZE

message would no longer work properly unless they were amended and

recompiled.

For backtesting, I've always used the IB realtime data which I collect in an

SQL database, and I've been perfectly happy with that. The fact is that

backtesting is very much only an approximation, especially if your strategy

uses any kind of limit orders: with market or stop orders you know that your

order would have been filled, but you can't be sure of what price you would

have got, whereas with limit orders you know at least the worst price you

would have got but you can't be sure you would have been filled at all

(unless price actually goes beyond your limit) – and your strategy might

behave quite differently subsequently depending on whether an order was

filled or not.

As an alternative to using the collected data for backtesting, one could

instead use the historical data supplied by IB – I believe that it reflects

the complete information sent by the exchange. The problem here is that IB's

historical data is not always complete, even for periods where the realtime

data feed was ok. One could also try a hybrid approach where you collect the

realtime data and then update it later on with whatever historical data you

can get from IB. But in my view, this is overkill.

Richard

[Q] What are the differences between Level I and Level II data ?

[A] found in this thread

In reference to our previous thread counting market data events, I did a

little analysis counting the number of Level 1 updates vs Level 2 updates at

the inside market.

My futures trading platform relies upon the current Bid / Ask price, which I

normally derive from the Level 1 updates.

My analysis shows, however, that Level 2 (DOM, Market Depth, the Book)

updates Position 0, which is the inside market, at about 5 times that rate.

So there might be about 200 or so updates of the Bid size in a minute, but

there might be 1000 or so updates of the inside Bid on the DOM.

On the face of it, that is more current information perhaps than the Level 1

update. Hmmmmmmmmmmm…… πŸ™‚

So I think I need to review / upgrade my code to possibly use this more

frequent DOM update value as my most "current" value for the (derived) Level

1 Bid or Ask price. Whaddya think?

[Q] Which request ID should I use?

[A] by Kurt

The people who can answer your question are probably the people who simply

tried every possibility. I did once, but not for scanner. But as I recall

each type of request has its own space for request id's. So you probably

have a lot of freedom. But at the same time you might as well choose id's

that are more convenient for your own tracking.

For that purpose I use a unique id for each request, globally across all

request types *except* for orders, for which I use a method specific to

orders which disregards whether the same request is used for other request

types. But that's just an accident of development.

Some others use unique request id's in different ranges for different

request types, so they can tell directly from the request id what kind of

request it was associated with.

So you can do what you like. But as to your original problem I have no

idea, except I recall people have had request id uniqueness problems related

to multi-thread coding problems, critical section issues or just not

incrementing at the right point in their code. No idea whether that applies

to you.

-Kurt

[Q] How deep can I go in history?

[A] by Dmitry

Usually it is only -1 year back, but if you pay (or trade) more you can reach up to -4 years, which brings you somewhere to Jan-2010 (quite far from desired 2000:(

a quote from this page:

  • Historical data requests can go back one full calendar year or more, depending on the number of concurrent real-time market data lines:

Number of Market Data Lines

Historical Data

Request Limit

Less than 499

One year

500 – 749

Two years

750 – 999

Three years

1000

Four years

 

Market data lines can be increased based on monthly commission amounts, amount of equity and Quote Booster subscriptions.

[A2] by souqMate

That table for hist data is only for stocks are FX.

Futures contracts are removed from IB's db 2 years after they've settled.

So even if you paid enough fees, you couldn't get more than 2 years.

Options (on futures or stocks) are removed right after they've settled.

souqMate

[Q] Bug in historicals? Price but no volume and other historical “holes”!

[A] by Kurt

What is missing from historical data can be for example particular bar sizes

for particular instruments, particular values of the "what" parameter for

particular instruments, etc., like the GLD example I cited. I take it that

option implied volatility is never available for GLD.

If you work with it long enough you will find that IB's "offering" is full

of holes. Not holes in time, but holes in the consistency of the offering.

There is no doc for what you will be able to get or what is not available.

IB either provides it or they don't. In many cases there is no error

message, just an absense of data.

-Kurt

[A2] by Ron Hinchley

With stocks missing data happens when no trades take place. I download 1-second data sometimes and even some NASDAQ-100 stocks have 1-6 seconds missing at the start of the day and sometimes an hour. When I run nightly I go back into my SQL two weeks looking for holes, then re-request the missing data if anything is longer than 5 seconds in a half hour. The data seems to stay missing but I don't check. Start of trading has often has an initial gap 1-5 seconds.

–Ron

[Q] Options historical data?

[A] by Kurt

I think IB does not offer 1-day data for options. I get the same result as

you. However I get data when I request 1-hour bars. However that won't

take you back a year. It looks like I can go back about 2 months. TWS

charts behave similarly.

Welcome to IB historical data. Well maybe you're not new, can't tell for

sure from your handle. (I wish people would sign their names, especially

people who use several email addresses. It is hard enough already.)

-Kurt

[Q] How to store historical data?

[A] by Dmitry (from this thread)

Added on 25-May-2017

Alex,

You can ask for 2000 1sec bars in one request (covers 33.3minutes). Simply advancing next request by some fixed amount of time won't work because when you request get close to the market open time then IB would send you back 2000 bars where some of them from one day and some of them from previous day!!  More over, if you request was close to some weekend/holiday (say another endDateTime happen to be 9:40am on Monday) then some of 2000 bars will belong to Monday while some to Friday around 3:45pm-4:00pm.

To get continuous history for N days you basically you need to do the following 3 steps:

  1) send reqHistoricalData with endDateTime pointing to the end of the period of your interest

  2) upon getting 2000 bars (historicalDataEnd()) you'd get oldest bar:

    and figure out if you have enough (i.e. if oldest bar's timestamp is older that the start of the time range you're trying to get)

  3) if not enough, then you send another reqHistoricalData() with endDateTime argument set to your oldest bar timestamp from step #2, and you loop to step#2

The above 3 steps would create nice and maximumly continuous set of bars with no overlaps between your requests. It would be good idea to store each complete 2000 bars in persistent storage (say DB), so if you happen to be interested in the historical data around that time it is available for reuse and you don't have to request it all over again, but there's more work to add persistent storage to your historical bars. I'll give a glimpse of potential problems:

[how you know if your stored 1-week worth of data is continuous and complete?]

Say you want to replay your backtest against 1 week of data. For simplicity sake – you have no data stored previously for given instrument. You loop through steps 1-2-3 (with 10sec sleep between requests to avoid hitting rate limits). One RTH=true day would be 6.5*60*60 = 23400.0 bars per trading day, which correspond to 23400 / 2000.0 = 11.7 historical requests and with 10sec gap in between requests it will be about 120sec = 2 minutes to download 1 day for 1 instrument (or twice as fast if you download your history simultaneously from paper and real accounts running on the same box on different ports;)

One week of data will take roughly 5 x 2 = 10 minutes. So 10+ minutes later your loop is over and your backtest fires many times and you're happy.

Next day after your platform restart you want to continue to play with exact same time range to test few more ideas, but now the problem is: how does your platform know if you have complete set for requested 1-week period or does it have any uncovered gaps? You can not rely on exact number of bars per day, because sometimes it will be actually less that 23.4K bars! Also sometimes there are halves of trading days with early session close or sometimes there are long weekends due to some holidays and the whole day might be missing, but 1-week is still continuous and there's no need to request anything!  Waiting another 10 minutes just to make sure that all 1-week stored bars are same 1-week bars available from IB does not make sense (and btw, sometimes it will be adjusted on IB end so it will be not exactly the same set of 1-week bars!-))

[how you properly merge existing "continuum(s)" with new data?]

But later on you decide you need 2 weeks of data instead of 1 week and it happen to overlap 1 week of data already stored in your (see previous step). Or in more complex cases you want larger period (say 1 year of data), which overlaps with few existing sparse weeks inside that larder period here and there. How would you solve the challenge of "not re-requesting IB for the data you already have?". As you can see for those cases simple steps 1-2-3 might turn into something more complex.

One of the possible simple solutions would be to mark your stored bars up to "complete day" precision. Then for each instrument/contract apart from just "bars" table you'd have to add another table with the only column: "complete days" and then while you're looping 1-2-3 steps backwards in time you can always check if you oldest bar happen to accidentally hit one of your "complete days" territory and then you trim 2000 requests to avoid overlap/duplicates in DB, store that trimmed subset and then fast-forward down to the morning of the "complete day" you've just hit (or even skip several days if they go one by one in a row and they all in "complete" state).

In this solution if you have 5.5 days of history stored 5 days will be marked as "complete" and 0.5 day will be not marked, thus next time the wider history range is requested you'd discard any incomplete data and would have to re-request that day over again (until it is complete).

Other approach that I implemented with more granular visibility into "completeness" would be to have 2nd table "continuum" with 2 columns for timestamps: "from" and "to". Each time you got 2000 bars – it is already something reliably continuous and you'd store 2000 bars along with new record into "continuum" table with "from" and "to" pointing into start and end of those 2000bars.  As you go with more complete historical data requests you just updating existing record for your continuum and by the end of retrieving 1-week period you have 23400 * 5 = 117000 bars saved in one table plus 1 record in your "continuum" table pointing into start time used for 1st request (can be anything, like even "midnight" btw) and end of that week (pointing to the oldest bar timestamp, because you can't use "midnight" here because you never know what data out there beyond your last requested range unless you commit to stay inside RTH=true for the rest of your life:). Then you work out the logic of merging 2 continuums as soon as they start to overlap, and that is it!  Now not a single request is waisted. Simple like that!-)

Cheers,

Dmitry

[Q] reqHistoricalData: Volume vs Count?

by Jason Jones <jasonjonesa11@hotmail.com> View/Reply Online (#42484)

Added on 27-May-2019

Can someone please explain the difference between the Volume and Count values that are returned in the Bar data after a reqHistoricalData?

IB's gives the following definitions for these properties:

Volume: The bar's traded volume

Count: The number of trades during the bar's timespan

I think of volume as "the total quantity of traded shares" which I would expect to be the same as the "number of trades".  But, the Volume and Count values are often different.

Here are some sample values returned for the Volume and Count after I requested 1 second historical data:

Volume, Count

72, 44

0, 0

2, 1

1, 1

46, 25

304, 135

20, 1

113, 51

3, 3

[A] by rwk View/Reply Online (#42485)

Back in the old days (i.e. 20th century), volume was usually not available until after the close or even the next morning.  Some traders followed "tick volume" whereby each trade caused the count to be incremented by one regardless of the number of contracts.  Some traders regarded tick volume as worthless, but others considered it [slightly] better than nothing.  Once realtime reporting of volume became available, it was often referred to as "live volume" to distinguish it from tick volume.  I doubt that tick volume gets used much anymore, especially in stocks.

(update by Nick): The count is trades, not orders. One trade (a buy and a sell) of 100 shares is  volume of 100 and count of 1.

[Q] How to backtest?

[A] by Kevin Zhu via Groups.Io <Z_kvn=yahoo.com@groups.io> View/Reply Online (#1531)

Added on 06-Jun-2018

Try Backtrader. Best pure Python backtesting open source solution I've found so far, excellent documentation.

I use insync to download data to feed into backtrader.

https://github.com/backtrader/backtrader

https://www.backtrader.com/

[Q] high, low, close fields in tickPrice ?

[A] by vishruth

My Qn to IB: In real time data access, what is the significance of high and low ? I should be just getting Bid, Ask and Last / Close only. Is that not right?

IB:

high – high price for the day

low – low price for the day

Above, they are mainly used for information. Normally, users obtain the information for trading analysis.

My Qn to IB: Also, what is the difference between last and close price returned by this API call ?

IB:

Last – The last traded price at which the contract traded

Close – Yesterday's closing price.

[Q] What is the sequence of tickSize() tickPrice events?

"If we get a size event then price event, then size event,

do we know whether the price should be associated with

the preceding size event or the subsequent price event?"

see also relevant: [Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] Building live Trades

[A0] From IB's discussion board:

We only send down changes and the order for which we check is:

Bid, Ask, Last, Bid Size, Ask Size, Last Size, High, Low, Volume, Close

So, in answer to your question the price event should be associated

with the subsequent NEXT size event (as price is sent before

size)

PRICE IS SENT BEFORE SIZE

[A1] by Dmitry:

I’d rather suggest to read IB API sources to see how and when API sends these events to your code. It’s more clear and its only about 100Kbyte+ c++ cource code to read!

[A2] blackbox_tws_api blackbox_twsapi@btinternet.com>

you can see what is going on under the hood with the EReader code.

The latest version will be >= 3 (currently version 7), so a price 'tick' sent from the IB server has the following information:

1. version of TICK_PRICE (>=3)

2. tickerId

3. tickType

4. price

5. size

6. canAutoExecute

Thus, the price tick received by the API has both price and volume information.  The API sequentially calls the tick_price and tick_size methods (note that it is impossible for the API to call tick_size again in between these two method calls, so the tick_size information immediately following tick_price is always related to that tick price).  

If the price has not changed, IB conserves bandwidth by only sending the current size information.  The size 'tick' has the following data:

1. version of TICK_SIZE

2. tickerId

3. tickType

4. size

Any size tick will thus relate to this previous price tick price of the same type (e.g. bid).

As you can see from the code below, the reader parses the tick data in lines 2-13 and then calls the tickPrice method on line 14.

After returning from that method call, it checks whether the TickType it read from line 4 was a price tick (1-Bid, 2-Ask, 4-Trade) or something else.  If it was a price tick, the EReader then calls the TICK_SIZE function on line 30.

As you can see from the code, the SAME tick read from EReade calls both the TICK_PRICE and the TICK_SIZE methods.

For example, this is the raw data for several ticks received for SPY using edemo (given that I need streaming data on a Sunday).

msgId                                    1                1                2                1                2

version                        6                6                6                6                6

tickerId                        1                1                1                1                1

tickType                        4                2                0                2                3

size                                2                32                48                30                30

price                                188.88        185.95                        185.75        

canAutoExecute        False        True                                True

msgId:          1=TICK_PRICE, 2=TICK_SIZE

TickType:        0=bid size, 1=bid price, 2=ask price, 3=ask size, 4=last price, 5=last size, 8=volume

As you can see, the data received by TWS/IBG is either a size tick or a size and price tick together.

I hope this helps to clarify what is actually happening with the tick price and tick size data.

 -BB

  1         case TICK_PRICE: {

  2               int version = readInt();

  3               int tickerId = readInt();

  4               int tickType = readInt();

  5               double price = readDouble();

  6               int size = 0;

  7               if( version >= 2) {

  8                   size = readInt();

  9               }

 10               int canAutoExecute = 0;

 11               if (version >= 3) {

 12                   canAutoExecute = readInt();

 13               }

 14               eWrapper().tickPrice( tickerId, tickType, price, canAutoExecute);

 15

 16               if( version >= 2) {

 17                   int sizeTickType = -1 ; // not a tick

 18                   switch (tickType) {

 19                       case 1: // BID

 20                           sizeTickType = 0 ; // BID_SIZE

 21                           break ;

 22                       case 2: // ASK

 23                           sizeTickType = 3 ; // ASK_SIZE

 24                           break ;

 25                       case 4: // LAST

 26                           sizeTickType = 5 ; // LAST_SIZE

 27                           break ;

 28                   }

 29                   if (sizeTickType != -1) {

 30                       eWrapper().tickSize( tickerId, sizeTickType, size);

 31                   }

 32               }

 33               break;

 34           }

 35           case TICK_SIZE: {

 36               int version = readInt();

 37               int tickerId = readInt();

 38               int tickType = readInt();

 39               int size = readInt();

 40

 41               eWrapper().tickSize( tickerId, tickType, size);

 42               break;

 43           }

[Q] I get negative and zero quotes! wtf?

[A] by Kurt

You should probably ignore quotes that are zero or negative,

unless you are trading combos. Your code should probably behave as if the

quote was not received, unless your logic depends on detecting absent

quotes, but even if so it is not a bad thing to try for diagnostic purposes.

Presumably if a negative quote causes a problem, and you ignore negative

quotes, your problem will go away. If so, then it does NOT sound like an

issue re the rate of requests. I think it is very unlikely that the rate of

your requests would *cause* IB to transmit zero or negative quotes.

Negative and zero quotes are a "valid" condition in the IB quote

stream. It is the normal state after-hours, and in some cases also when

liquidity disappears during trading hours (not unusual with US option

combos). You probably want to look at the size field as well to help

disambiguate these conditions if you are trading option combos, since in

certain cases a 0 or -1 price on a combo could be a real price. (But the

bid and ask will not *both* be -1.)

Also if you log your requests and request id's then you might well see

immediately whether you are transmitting more than request with the same id.

If you see this then you know you have a bug. Of course logging (especially

to an on-screen log window) might slow your data rate and hide a problem.

-Kurt

One correction. You can have negative bid *and* asks. I am writing some altos that scans options and very often on the way far away options there will be zero and minus one values. The minus one values annoy me to no ends as tws values them and all of the sudden I made or lost several million dollars on my currency future option positions.

Regards

Christian Gross

[Q] How to get all strike prices for a given ticker & expiry?

[A]

> First you need to invoke reqContractDetailEx(…)

> to obtain all related option products

> When invoking reqContractDetailEx(…) then you leave the one of the

> contract parameters empty, the empty attribute acts as a wildcard.

> Therefore, it will return all the valid combination products for

> that specified symbol. For example, I want to obtain all the valid

> strikes for IBM July option. In this particular case, I leave the

> strike field under contract object set to zero. Then this should

> return all the valid strikes for the month of July. See example

> parameter below:

>

> Symbol = IBM

> SecType = OPT

> Expiry = 201007

> Strike = 0

> Exchange = SMART

> Currency = USD

>

> The information delivers by ContractDetail(…) event.

>

> Raymund

> IB Technical Support

[Q] No security definition has been found.

[A] by blackbox_tws_api <blackbox_twsapi@…>

Always start with the IB Contract Information Center to ensure you have the relevant fields for your chosen contract:

http://www1.interactivebrokers.ch/contract_info/v3.8/index.php

Once you located the relevant contract, use the conid to request contract details.  You can then determine what corrections need to be made.

For your request below, there is also the potential issue of the expiration date.  In general, IB does not generally provide historical data more than one year old.

Other corrections:

Hide message history

contract.m_symbol="QQQ";

contract.m_secType="OPT";

contract.m_expiry= "20131018";                << Example using more recent contract, in YYYYMMDD format

contract.m_strike= 42;

contract.m_right = "C";

contract.m_exchange="CBOE";                << "SMART" will also probably work, depending on your requirements

Using the above and some intraday bar size settings (e.g. duration = "1 D", barSizeSetting = "5 mins"), I get an error of No market data permissions for CBOE OPT rather than the previous No security definition has been found.

Hope this is useful for you.

 -BB

[Q] reqMarketData(): For most stocks I get the last prices, however for some stocks I only get bid.

[A] by Kurt

Well that *does* happen some times. Particularly on very-low-volume stocks

I've seen the bid or ask disappear and then return a couple seconds later.

But it is pretty unusual in my experience and I don't see it in US stocks

during regular trading hours. I believe I see it during regular trading

ours on some US ETF option (native) spreads.

However, I think there should always be a "Last" except if there has been no

trading yet that day, on that issue, i.e. if the volume is 0 I think there

will be no Last. TWS itself will display the closing price with a "C"

appended in that circumstance, but the API does not do this.

Of course you can have a real scenario in which there is no Bid because no

one is buying–there is no market, as in the case of options near expiry

that are far out of the money.

-Kurt

[Q] reqMarketData(): sometimes getting back tickPrice events with a price of -1

[A] by Kurt

I have the impression even IB does not know what values will be returned in

the price field. I have variously seen -.99 and -1.0 and 0 and

no-value-transmitted after hours. IB says they pass on data directly from

the exchange (generically speaking) and have no control over it. So you

will get all kinds of nonsense after hours while the exchanges dick around

with their systems. You have to determine yourself whether a price is out

of range. Even during trading hours exchanges may post non-liquid bid/ask

values that look very good but won't execute. For this reason some people

restrict their market data to a single high-volume exchange rather than

using @SMART for equities, even though they still place trades using SMART.

For FX that doesn't apply.

For combos a negative price can (sometimes) be valid, so always good to

examine quotes on all the legs as a reality check. The bid/ask on a native

combo should always be inside what you would compute by adding/subtracting

the quotes you get for the legs, remembering to swap bid/ask when for

negative weights (ratios).

The market data feed you get from IB is kind of a free-for-all. There are

no rules. A price is either real or it isn't. If it's not, it won't

execute. πŸ˜‰

-Kurt

[A2] by climbon6

IDEALPRO's hours are 17:15 – 17:00 (ET) Sunday-Friday.

The market was closed when you got the Bid price = -1.0

http://interactivebrokers.com/en/trading/exchanges.php?exch=ibfxpro&showcategories=FX&ib_entity=llc

[Q] What is conId?

[A] by Kurt

conId is IB's way of identifying a contract. Think of it as IB's own cusip.

At least AFAIK it has no relation to any one else's system.

There are many contexts in which conId is the only method to identify a

contract. If you specify a contract via the Contract structure, you will

sometimes need to request contract details just to get the conId so you can

use it where it is required. One such place is when building combos. Look

at ComboLeg and you will see there is no other way to specify a leg except

by the conId.

I can't remember whether there are any other contexts that absolutely

*require* conId.

However you might possibly want to use it for persistence of contracts in

use from one invocation of your app to the next, since even if the symbol

for a contract is changed, the conId will not. (This will cease to be the

problem for US options that it used to be, but it can happen with stocks as

well, much more rarely.)

-Kurt

[A2] by Kurt (acutally answer taken from other thread, but this particular one is more about conId):

To the OP's question very likely your point is basically correct. If

setting contract id to 0 worked fine, then there is no (immediate) need to

specify it. (Perhaps now IB has started checking that you don't specify

BOTH the contract id and other Contract fields?)

In any case it would be interesting if the OP would give a full example of a

case that fails, so we can take a look at it.

However I have to disagree re the contract id being only relevant for

combos.

One helpful technique to avoid problems with ambiguous contracts and

Contract specifications that are rejected by requestMarketData or

placeOrder, is to specify the full Contract fields only in

requestContractDetails, obtain the contract id from that, and then use that

contract id (without the other Contract fields) to request market data and

historical data and to place orders.

Various people have been running into problems recently with certain

Contract specifications that used to work and/or that still work with

requestContractDetails but not with requestMarketData or placeOrder. Using

the Contract Id (instead of the other Contract fields) for as many things as

possible avoids this kind of issue, and avoids strange surprises when IB

periodically messes with how placeOrder (or whatever) interprets the other

contract fields.

Using the Contract id also gets you through any symbol changes that might

occur overnight (used to happen regularly with options prior to OSI), which

might find you unable to close the position you opened yesterday using the

same Contract fields.

And of course, as you said, you are forced to use it when specifying combos,

but it is relevant for many other reasons.

-Kurt

[Q] Volume looks incorrect?

[A] by souqmate@yahoo.com

volumes for US stocks have an implicit factor 100 in TWS and in API; true for historical data and for real time data (bidSize or askSize).

[Q] How to get indices historical prices?

[A] by orionn2 <no_reply@yahoogroups.com>

indices don't have bid nor ask prices, hence no (bid ask) midpoint data is available, try using TRADES instead which will give index prices.

[Q] How to get realtime P&L?

[A] by Kurt

> Hi all:

>

> I'm having a very difficult time trying to get realized and unrealized P&L

> from IB. After I establish a connection I call reqAccountUpdates(true, "");

> on the socket, the updateAccountValue() callback method gets called

> frequently, but updatePortfolio() gets called almost never. Is there

> something I'm missing?

I'm not watching that data carefully and since it is the weekend I can't

really go try it. So I'm not sure what your issue is.

My recollection is these things get called periodically, maybe every few

seconds. That's not often enough for my liking. (I think you also get

updates after trades more quickly.)

Personally I like to see P&L in realtime and these updates are not realtime.

For now I use the IB-provided position and cost basis, but I ignore the rest

and compute the P&L myself from the position, cost basis, and tick data.

TWS itself could have done the same, but does not. It seems idiotic to

display delayed P&L values when live data is just sitting there being

ignored. The only reason for TWS to use the delayed P&L data would be as a

fallback if you are not getting IB tick data. But if you are writing an API

app then you don't have that excuse–you can use tick data wherever you get

it from to compute your P&L.

Also if you do it yourself you get to set your own policy on how P&L should

be calculated. I believe IB uses the last price clamped to within the

bid/ask range. That's reasonable perhaps as a general approach but may not

be a safe enough indicator for some people's liking, since the "Last" price

may never happen again. Worse yet if it is delayed a few seconds!

-Kurt

[Q] reqRealTimeBars() – "whatToShow" string?

by someone765@yahoo.com

Recently someone asked if the "whatToShow" string could have multiple entries or would be single valued: TRADES, BID, ASK, MIDPOINT".  Kurt kindly pointed out that the entry had to be single valued.

Do I correctly assume that the realtime bar data will continue with what was last set in this field, until overridden by a new reqRealTimeBars() command? And can a person obtain the trades and bid and ask data successfully during just one 5 second bar, but 3 requests will have to be rapidly made within this interval specifying such to obtain the data successfully?

In other words, can we obtain all the data for the 5 sec bars for a stock and how is this done?

Randall

[A] by Ed news1000@edshome.net

Yes, you request once and then get a reply automatically every 5 seconds.

Each request has its own id so you can have multiple requests concurrently.

You are still bound by the data limits (100 simultaneous market data requests by default) but I'm not sure exactly how this applies to realtime bars. If you are just monitoring a few symbols it will be fine but if you are looking for thousands then it will be a problem.

–Ed

[A2] by Kurt Bigler kkb@breathsense.com

Each request is a subscription, in a sense just like reqMktData.  If you want updates for TRADES, BID, and ASK, you make 3 requests.  People usually make realtime bar requests to get ongoing updates, not for a single bar.  You have no guarantee that your first results for all 3 will start in the same bar period.  But they will continue and you will end up getting ongoing updates from all 3.

-Kurt

[Q] reqRealTimeBars() – only 5sec bars available? Partial bars?

[A] … you can calculate them from 5-second bars. You can calculate other bars from 5sec bars, or you can calculate bars from market data, or you can simply request historical bars periodically, which even lets you get partial bars in progress when you request endDateTime="".

-Kurt

[Q] tickPrice(): canAutoExecute field – wtf?

The documentation on this flag is as follows:

Specifies whether the price tick is available for automatic execution. Possible values are:

• 0 = not eligible for automatic execution

• 1 = eligible for automatic execution

By 'automatic execution' do they mean 'placing an order'? If so, what is the point of sending ticks which are not eligible?

[A] src

My understanding of autoexecution is that it's a hold-over field from the pre-hybrid days…. on the NYSE they used to have two exeuction platforms, one that was manual through the floor specialists and one that was automatic through a computer system… the computer-based orders would only handle a certain volume, so orders that could be autoexecuted and hit on the automated system were tagged for the trader.

However in 2007 they upgraded their automated system and moved everything that way, so I think 99-100% of orders posted to the book should be able to be executed automatically (no human intervention required to hit the bid or take the offer).

So I can ignore it as legacy?

yeah I wouldn't worry about it.

[Q] When requesting historical data I often got “Invalid step”

 What should I do?

[A] "Invalid step" really means invalid combination of step (bar size) and request duration.  Usually you know what bar size you want, without question, and will use whatever is the maximum duration that works, so "Invalid duration for bar size" would be more sensible.  But IB seems to think consistently the other way on this, both in the API errors and in their documentation.

[see table on the next page]


I have always referred to the following table which has been posted to this list at various times, and as a result I have never gotten the Invalid step error.

    http://www.tradingsoftwarelab.com/jtwsdump/table.html

by Kurt Bigler kkb@breathsense.com via yahoogroups.com

mark duration values BLUE which yield max number of bars (multiple marked in one raw – means yield same results aka “same good”)

\ duration

   \

     \

        \

bar size\

Y

M

W

D

S

1 year

86400

3 month

86400

1 month

5

86400

1 week

5

12

52

60

86400

1 day

5

12

52

60

86400

1 hour

1

4

34

86400

30 min

1

4

34

86400

15 min

2

20

86400

5 min

10

86400

3 min

10

86400

2 min

10

86400

1 min

10

86400

30 sec

1

86400

15 sec

30000

5 sec

10000

1 sec

2000

Return to the jTWSdump homepage.

Example: I want to pull bar size = 30 min, and I’m going to specify duration in “D”ays, which value for duration should I use (i.e. what is tha max allowed one?). The table yelds “34”, so the answer is “34 D” for your duration parameter for reqHistoryData().

Note, 34 trading days is > than 34 calendar days.

Note: 34 trading days is > than specifying “1 W” or “86400 S”.

(todo: run all those queries and add into the table actual number of bars returned for cases where bar size != duration units, mark duration values BLUE which yield max number of bars)

also: Riggster: I didn't realize that when you specify a barsize of 1 – 15 seconds, you have to specify the durationStr as seconds, not days, so you could use '23400 S', but not '1 D', even though there are 23400 seconds in 1 day.

[A2] by orionn2 (18-Dec-2015):

I have updated the table in the jTWSdump website: http://www.tradingsoftwarelab.com/jtwsdump/table.html

showing increased limits of 5 Y (5 years) for 1M (1 month), 1W (1 week) and 1 day bars and also 10 D (10 days) for 5 mins, 3 mins, 2 mins and 1 min bars. These limits are for a single data request – much more data can be downloaded by setting the End Date/Time in another API request to a date before 5 years from now. In jTWSdump this can be easily managed by using the "Number of Queries" field.

Thanks to Richard L King for the heads-up.

also see next Q/A about bar size settings:

[Q] Valid Duration and Bar Size Settings for Historical Data Requests

[A] quote from this page:

The following table lists valid duration and bar size settings for API historical data requests. Please note that these are only guidelines.

Duration

Bar Size

DURATION

1 Y

BAR SIZE

1 day

6 M

1 day

3 M

1 day

1 M

1 day, 1 hour

1 W

1 day, 1 hour, 30 mins, 15 mins

2 D

1 hour, 30 mins, 15 mins, 3 mins, 2 mins, 1 min

1 D

1 hour, 30 mins, 15 mins, 5 mins, 3 mins, 2 mins, 1 min, 30 secs

14400 S (4 hrs)

1 hour, 30 mins, 15 mins, 5 mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs

7200 S (2 hrs)

1 hour, 30 mins, 15 mins, 5mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs

3600 S (1 hr)

15 mins, 5 mins, 3 mins,  2 mins, 1 min, 30 secs, 15 secs, 5 secs,

1800 S (30 mins)

15 mins, 5 mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs

960 S (15 mins.)

5 mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs

300 S (5 mins)

3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs

60 S ( 1 min)

30 secs, 15 secs, 5 secs, 1 secs

[Q] Is it possible to develop TWS like depth book using reqMktDepth() call?

by hyeungsf@yahoo.com

[A] by barnstws <no_reply@yahoogroups.com>

Try specifying the primary exchange field. reqContractDetails will help you get the exact parameters. Forex is free and easier to see if you are doing it right.

[Q] What happens when I do reqMktData() afterhours?

[A] by Dmitry

In case of successful subscription you’ll get one tickPrice() event with tickType=CLOSE (basically latest price (no volume))

OR

You’ll get either an error() event (if your contract description is somewhat broken or you set same tickerId as for your previous subscription or other error)

Examples:

[successfull afterhours subscription]

05:18:20 PosixTestClient.cpp:739 reqMktData():

05:18:20 PosixTestClient.cpp:746 requesting market data for contract:

             conId: 0

            symbol: FB

           secType: STK

            expiry:

            strike: 0

             right:

        multiplier:

          exchange: SMART

   primaryExchange:

          currency: USD

       localSymbol:

      tradingClass:

    includeExpired: no

         secIdType:

             secId:

  comboLegsDescrip:

         comboLegs: no legs found

05:18:20 PosixTestClient.cpp:247 tickPrice():

         tickerId: 0

         TickType: CLOSE (9)

         price: 55.44

         canAutoExecute: 0

[bad afterhours subscription attempt]

05:23:15 PosixTestClient.cpp:739 reqMktData():

05:23:15 PosixTestClient.cpp:746 requesting market data for contract:

             conId: 0

            symbol: FB

           secType: STK

            expiry:

            strike: 0

             right:

        multiplier:

          exchange: SMART

   primaryExchange:

          currency: CAD <–– this makes it bad

       localSymbol:

      tradingClass:

    includeExpired: no

         secIdType:

             secId:

  comboLegsDescrip:

         comboLegs: no legs found

05:23:15 PosixTestClient.cpp:232 error():

         id: 0

         errorCode: 200

         errorString: No security definition has been found for the request

or some other errors, like:

         errorCode: 322

         errorString: Error processing request:-'vc' : cause – Duplicate ticker id

[Q] IB's own historical data corrections (history of the history:)

Added on 27-Apr-2015

[A] by Kurt (found here)

Although I'd heard it mentioned (even by IB I believe) I had never looked at

what changes over time actually occur in IB's historical data, i.e. the

history of the history.

Today I observed the following in 1-min USD.CHF BID_ASK bars. The times

shown are US Pacific Time.

Some time on 5/30 I downloaded this bar from the day before:

20110529 22:52:00 O 0.85215 H 0.85225 L 0.8521 C 0.85225

but today (6/5) when I download the same bar it had changed to:

20110529 22:52:00 O 0.85215 H 0.85215 L 0.8521 C 0.85215

^ ^

I have not gotten around to comparing this to 1-second data to see how it

relates. I never acquired the data I would have needed to do that in this

case.

(ASIDE: These are BID_ASK bars, so open and close don't have the usual

meaning. However they are supposed to be time-weighted averages, and as

such they are suspect being so exact. I believe it may have come up that IB

rounds their averages to the nearest tick, which of course makes the average

less accurate. I think I had observed this but also I had some other

problems that may have occluded accurate observation, since I was

unknowingly rounding my own archived data (to what the C format %g

produces). Since fixing that I have observed that OPTION_IMPLIED_VOLATILITY

historical data does NOT have the time-weighted averages rounded. But of

course there is no minTick value that applies to volatility.)

I also have seen evidence of some BID and ASK bars disappearing from the

record. These may have been after-hours bars. I don't have the info handy

on this right now.

The main point is that I have now seen an actual change in the history of

historical data, whereas previously this was just conjecture for me. I

don't know exactly when the change noted above was made because it was a

week before I checked it again.

I'll eventually look at this in more detail, but I'm wondering what others

do with respect to this issue.

One possible strategy is to wait a certain period of time (several days?)

for the "majority" of history changes to settle out, before downloading.

I'm curious how many people intentionally do NOT download up to the current

moment for this reason. If so, how long do you wait for data to stabilize?

If you answer, please indicate what security or type of security you are

working with.

Thanks for any thoughts.

-Kurt


TWS Scanners

[Q] Where do I begin with scanners?

[A] Look at the Market Scanner item on the Market Data tab of the ApiDemo program. This will give you a good start.

[Q] Biggest gainers since open between $5 and $10?

Trying to return the biggest gainers since open between $5 and $10.

 I have successfully returned this in TWS, but I cannot get the exact same result in the API. Below is my code, then the TWS scanner file that I am trying to copy.  What am I missing here?  Thanks!

Creating scanner:

ScannerSubscription scan = new ScannerSubscription();

            scan.scanCode("TOP_OPEN_PERC_GAIN");

            scan.instrument("STK");

            scan.locationCode("STK.US.MAJOR");

            scan.belowPrice(10);

            scan.numberOfRows(10);

            

            m_client.reqScannerSubscription(0, scan);

Output method:

public void scannerData(int reqId, int rank, ContractDetails contractDetails, String distance, String benchmark, String projection, String legsStr) {

            System.out.println(contractDetails.m_marketName + " " + reqId + " " + rank + " " + distance + " " + benchmark + " " + projection + " " + legsStr);

    }

The TWS file:

<?xml version="1.0" encoding="UTF-8"?>

<ScannerTemplate autoRefresh="true">

        <ScannerContent varName="content" version="0" watermarkWhenEmpty="false" pageId="1385043649" transferableParameter="true" locationText="STK.US.MAJOR" statsExchText="ALL" maxItems="-1" snapshot="-1" paused="false" scannerSent="true" chartDividerLocation="0">

                <ExtendedConfigurationMap varName="extendedConfiguration"/>

                <scannerName></scannerName>

                <Instrument varName="instrument" name="US Stocks" m_type="STK" filters="PRICE,PRICE_USD,VOLUME,VOLUME_USD,AVGVOLUME,AVGVOLUME_USD,HALTED,UNSHORTABLE,SHORTSALERESTRICTED,NUMSHARESSHORT,SHORTOFFLOATPERC,DAYSTOCOVER,NUMSHARESINSIDER,INSIDEROFFLOATPERC,NUMSHARESINSTITUTIONAL,INSTITUTIONALOFFLOATPERC,OPTVOLUME,AVGOPTVOLUME,OPTVOLUMEPCRATIO,IMPVOLAT,IMPVOLATOVERHIST,MKTCAP,MKTCAP_USD,STKTYPE,PILOT,DIVIB,DIVYIELDIB,NEXTDIVDATE,NEXTDIVAMOUNT,HISTDIVIB,HISTDIVYIELDIB,GROWTHRATE,PERATIO,QUICKRATIO,DIVYIELD,RETEQUITY,PRICE2BK,PRICE2TANBK,CHANGEPERC,AFTERHRSCHANGEPERC,CHANGEOPENPERC,OPENGAPPERC,PRICERANGE,IMBALANCE,IMBALANCEADVRATIOPERC,REGIMBALANCE,REGIMBALANCEADVRATIOPERC,AVGRATING,NUMRATINGS,AVGPRICETARGET,NUMPRICETARGETS,AVGTARGET2PRICERATIO,TRADECOUNT,TRADERATE,VOLUMERATE,SIC,INDEX_COMPARISON,FIRSTTRADEDATE,HASOPTIONS" group="STK.GLOBAL" shortName="US"/>

                <ScanType varName="parameter" displayName="Top % Gainers Since Open" scanCode="TOP_OPEN_PERC_GAIN" instruments="STK,STOCK.NA,STOCK.EU,STOCK.HK,FUT.US,FUT.HK,FUT.EU,FUT.NA" absoluteColumns="false" supportsSorting="true" respSizeLimit="#" snapshotSizeLimit="250" feature="PCT" searchDefault="false" access="allowed">

                        <Columns varName="columns">

                                <Column colId="55" name="Change Since Open" display="true" section="m">

                                        <displayType>DATA</displayType>

                                </Column>

                                <ColumnSetRef colId="0" name="BA" display="false">

                                        <displayType>DATA</displayType>

                                </ColumnSetRef>

                                <Column colId="48" name="Time in Scan" display="true">

                                        <displayType>DATA</displayType>

                                </Column>

                        </Columns>

                </ScanType>

                <SettingValueList varName="settingList">

                        <SettingValue settingCode="annualVolatility" value="true"/>

                </SettingValueList>

                <AdvancedFilter varName="filter">

                        <priceAbove>5</priceAbove>

                        <priceBelow>10</priceBelow>

                </AdvancedFilter>

        </ScannerContent>

        <ArString varName="fullySelectedNodesList">

                <String>STK.US.MAJOR</String>

        </ArString>

        <Columns varName="tickerColumns" type="jcolumn.TickerColumns">

                <ContractSingleDescColumn colId="99" width="176" alignment="2" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false"/>

                <ChangeSinceOpenColumn colId="55" width="172" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" showPercent="true"/>

                <LastPriceColumn colId="7" width="188" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" prohibitDynamicColors="false"/>

                <BidPriceColumn colId="4" width="188" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" prohibitDynamicColors="false"/>

                <AskPriceColumn colId="5" width="256" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" prohibitDynamicColors="false"/>

                <InScanTimeColumn colId="48" width="256" alignment="4" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false"/>

        </Columns>

</ScannerTemplate>

Oh, and when I create the scanner, there should be a line that says:

scan.abovePrice(5);

Idk why that didn't copy over.

Q by:

From: bgould96@aol.com

Sent: Sunday, November 24, 2013 11:19 AM

To: TWSAPI@yahoogroups.com

Subject: [TWS API] RE: Trouble getting API (Java) scanner to output the same data as the one in TWS

[A] by WeifeiL@gmail.com weifeil@gmail.com via yahoogroups.com

Two things:

1. there is a bug for API:

 

you need to add this one to the scan obj:

 

scan.averageOptionVolumeAbove( 0 );

 

 

2. For each scan, you need to specify a unique id. And this id will be associated to the scanner and not changeable before you restart your IB TWS/Gateway.

 

Good luck with the scanner functions.

 

Weifei

[A2] by supermanz@gmail.com via yahoogroups.com

I think you're looking at the wrong field in the result.  Try contractDetails.m_summary.m_symbol.

bgould96@aol.com via yahoogroups.com:

Ahhhh, that works!  Thank you so much!  Just purely for the sake of knowledge, what exactly is returned by contractDetails.m_marketName?

[Q] Give me simple example of working scanner request.

[A] you can get it from following communication (taken from here):

I have the code for connecting to TWS … it's been working for several years. Today, I tried to access the scanner … like this:

… connect to TWS

ScannerSubscription ss = new ScannerSubscription();

ss.numberOfRows(10);

ss.instrument("STK");

ss.locationCode("STK.NYSE");

ss.abovePrice(1.0);

ss.belowPrice(10.0);

ss.scanCode("TOP_PERC_LOSE");

ss.stockTypeFilter("STOCK");

toTws.reqScannerSubscription(id, ss);

… after scannerDataEnd is called, I can successfully cancel the scanner subscription

When I run the above I get:

error: id: 936, errorCode: 165, msg: Historical Market Data Service query message:no items retrieved

If I interchange 1.0 and 10.0 above, I get:

error: id: 936, errorCode: 162, msg: Historical Market Data Service error message:Invalid Price Filter

Clearly I'm missing something or I don't understand something. Can anyone provide any insights???

———————————— answer ———————

Hi Lewis,

From ScannerSubscription.java:

private double m_abovePrice = Double.MAX_VALUE;

private double m_belowPrice = Double.MAX_VALUE;

private int m_aboveVolume = Integer.MAX_VALUE;

private int m_averageOptionVolumeAbove = Integer.MAX_VALUE;

private double m_marketCapAbove = Double.MAX_VALUE;

private double m_marketCapBelow = Double.MAX_VALUE;

Set abovePrice, aboveVolume, and marketCapAbove (to Double/Integer.MIN_VALUE, for example), and I think your

problem will be fixed.

David

———————————— and final working scanner is:  ———————

Thanks for the help. It works now. Here's the code:

ScannerSubscription ss = new ScannerSubscription();

ss.numberOfRows(10);

ss.instrument("STK");

ss.locationCode("STK.NYSE");

ss.abovePrice(3.0);

ss.aboveVolume(0);

ss.marketCapAbove(100000000);

ss.averageOptionVolumeAbove(0); // had to have this!!

ss.scanCode("MOST_ACTIVE");

toTws.reqScannerSubscription(936, ss);

[Q] Can't get scannerSubscriptionOptions to work

Is anyone else using scanners?  They are so under documented!  I am doing bond scans which work fine in part.  For example (in C# calling J#):

                int nTickerID = 301;

                ScannerSubscription ss = new ScannerSubscription();

                ss.numberOfRows(10);

                ss.instrument("BOND");

                ss.locationCode("BOND.US");

                ss.scanCode("HIGH_BOND_NET_ASK_YIELD_ALL");

                ss.maturityDateAbove("20171001");

                ss.belowPrice(100.1);

                ss.moodyRatingAbove("BA3");

                ss.spRatingAbove("BB-");

             

                java.util.Vector opts = new java.util.Vector();

                opts.add(new TagValue("VARCPNRATE", "false"));

                engine.IBHandler.ClientSocket.reqScannerSubscription(nTickerID++, ss, opts);

I'm returned the 10 bonds I requested, except I can't get scannerSubscriptionOptions to work for me i.e. they are a mix for fixed and variable rate bonds.  I have studied the reqScannerParameters() XML to try to filter down more.  VARCPNRATE is supposed to work for BOND, but how?

Has anyone gotten scannerSubscriptionOptions  to work?  Is there a guide anywhere for these TagValue options?  Any hints are appreciated!

[A] by Josh (found here)

Added on 10-May-2017

Hi Harlan, unfortunately the API scanner is quite limited in available filters and scans as compared to the TWS Advanced Market Scanner. The reqScannerParameters has parameters which are available in the TWS scanner but not all are available from the API. The third parameter in reqScannerSubscription is used only by some third party applications to specify the origination of the request, otherwise its null. The available filters are simply the members of the ScannerSubscription object.

Josh

[Q] Note on contracts which may not be accessible through TWS when running scanners:

[A] by Josh View/Reply Online (#41572)

If you receive scan results in the TWS Advanced Market Scanner which are not available to the API, this can be because the results contain contracts which are not tradable through TWS. The TWS Advanced Market Scanner has a feature to automatically skip these results, whereas the API scan does not. I would suggest first double-checking what the TWS Advanced Scanner displays for the same scan if the scan code and filters are chosen to match exactly.

[Q] Market Scanner Pre/Post Market Hours:

[A] by David Armour dave.armour    at  gmail.com View/Reply Online (#42026)

OK – I figured out a solution with the scanner to get what I need for stocks that are gapping in the pre-market..

I use Top % Gainers/Losers (and not Close to Open)

I use Change % (and not Gap %) ensuring to set LESS THAN -X% when looking for losers and GREATER THAN X% when looking for gainers.

If I set the filter to the Change % I get the Gap % since the close of the previous day. This is working in Pre-Market which is what I needed.

I hope this helps anyone else struggling with pre-market scans.

  • To clarify, will the Top % Gainers/Losers scanner update pre/post hours, without any filters ?
  • Yes. Just don't use the After-Hours or Pre-Market Hours option which seems only to work when the market re-opens.

[A2] by Josh View/Reply Online (#41888):

On Sat, Mar 2, 2019 at 05:26 PM, Lightbrook Brokers wrote:

Will the callbacks for reqMktData (of tick type 4, last price) continue updating for pre/post hours?

Yes the results including last traded price will continue to update in pre/post regular trading hours.

I also tried Top % After Hours Gainers/Losers and that also returns nothing.

You might want to take a look at this post https://groups.io/g/twsapi/message/41570 and add a filter in the same way to limit the scan results further. If the scan results are instruments which can't be traded at IB they won't be returned to the API.

Basics for beginners

You should always try the corresponding functionality in TWS. If it

doesn't' work in TWS either, then it is not an API problem. (Kurt)

[Q] Vocabulary

[A]

ATS – automated trading system

BBO – Best bid offer (NBBO = National BBO) (RBBO = Regional BBO)

DOM – Depth of market

CFD – Contract for Difference

GAT – good-after-time (GAT) orders

MOC – market on close order (example: MOC order in an oca group)

              MOO and MOC work for U.S. stocks. Check the IB website for other markets.

              One catch for MOO orders is that you have to specify "OPG" for the time-in-force.

Also, these order types can not be canceled after a specific time. I

haven't done this is in a while but it used to be for stocks you could

not cancel a MOC after 3:50 pm NY time. Be sure to research all the

details before you place these order types.

MOO – market on open order

OCA – once-cancels-all

OCO – one-cancels-other (when one order executes it will automagically cancel another order:)

Note about OCA/OCO: by Richard L King from this thread

There is no guarantee that this will avoid the problem. Where two orders in an OCA group have fairly close prices, it is possible for both to be filled in a fast market. This is because when the first order is filled, IB cannot know to cancel the second order until it has received the fill notification for the first from the exchange, and it then has to transmit the cancellation to the exchange: during this time the second order could be filled.

This did actually happen to someone a few years ago, and they complained to IB, but IB (quite rightly) pointed out that they cannot make any guarantee that only one order in an OCA group will be filled.

A much safer approach is to simply use a single stop loss order, and adjust its price as required: it can be included as a child order when the entry order is submitted, and with a wide stop for safety. Once the entry order has filled, the trigger price for the stop loss order can be tightened as required, and subsequently adjusted as profit increases.

RTH – regular trading hours (like 9:30 to 16:00)

NBBO – Relative (a.k.a. Pegged-to-Primary) orders provide a means for traders to seek a more aggressive

price than the National Best Bid and Offer (NBBO)

VWAP – volume weighted average price

[Q] Limitations on quantity of symbols?

[A] from IB website

All customers initially receive 100 concurrent lines of real-time market data (which can be displayed in TWS or via the API) and always have a minimum of 100 lines of data. Customers can increase the allowance of simultaneous quotes windows by purchasing monthly Quote Booster packs.

[Q] Where can I find the API version info?

[A]

C:\Jts\API_VersionNum.txt

[Q] I’m about to post a question to IB API list. What should I do?

[A]

1. do basic google search for your question before asking your [Q]

2. search TWS API archives for some of keywords related to your question

3. sure, nothing found.. you’re writing your post to The Group

    – include the API type that you trying to use

    – try to include what exactly you’re sending to API

    – what replies (in form of callbacks) you receive?

        – are you logging all the possible API callbacks or may be you just missing one

   thus thinking “I did ABC, but nothing happened…”?

    – generally try to be precise (not vague)

    – try not to send more than 10 posts/day πŸ˜‰

     (Few thousands subscribers will have to read it. Have some respect:)

[Q] Number of hops to IB?

[A]

Has anyone else experienced HDMS market data connection issues since upgrading

to 906?

I couldn't get a stable connection so I downgraded back to 900.

Support is telling me I have too many hops in my tracert! Apparently the support

engineer thinks British Telecom will reduce them for me down to 15 (from 26) if

I complain, or else change ISP!

— Adam Hardy

I always used to ping gw1.ibllc.com

Right now I get about a 50 msec avg round trip from North Carolina, USA on

Road Runner Turbo over 14 hops.

— FutureScalper

others:

14 hops, 55ms here (16Mb comcast business class)

9 hops, 37ms (Small Co-Location facility in Maryland)

In relation to connection speed, just reading the other current thread, tracert

shows I am 25 hops from the server at 200ms – I'm in Europe – and the IB support

staff say this is not a guaranteed good connection πŸ™

Ping @ 95ms each.

[Q] Is IB real time data actually “real time data”?

[A] (investigate more)

IB's data is not "live" to begin with. it's series of snapshots with 5-30 ms intervals(don't remember exactly, you can check TWSAPIgroup on yahoo-there is very detailed explanation somewhere about how IB deliver the data). bottom line-it's not live and this is why IB's data usually doesn't lag at the open or during Fed announcements. so-there is no such thing as "fast market" for IB's data

— also —

You would not expect IB's ticks to match anyone else's. You are aware that

IB's ticks are sampled?

-Kurt

I don't keep the exact details of it in my head, but it is something like a

limit of 10 ticks per second that is imposed, with a higher rate I think for

certain types of instruments. (Someone posted definitive information on

this a few months ago but I can't find the message.)

-Kurt

— also —