Dmitry’s TWS API FAQ

Share this post

Dmitry’s TWS API FAQ

Last updated on: 12-Jun-2022

Table of contents:

About this FAQ

Purpose

Scope

Disclaimer

Link to the latest IB API documentation

TWS app related

[Q] How to turn off Security Code for TWS real account login?

[Q] How many market data lines one has subscribed to in TWS?

[Q] How to import Tickers from a File?

[Q] Link to the official server reset time?

[Q] How to check and solve connectivity issues affecting the Trader Workstation (TWS)

[Q] How to run 2 TWS on different machines (to access same account)?

[Q] How to optimize TWS Java VM performance? aka the "latest and greatest" JVM server settings

[Q] How to make TWS application running 24/7 automagically?

Optimal memory setting

[Q] How to set up IBController (3.2.0) + TWS (build 956)  on headless Ubuntu 16.04 LTS to run TWO accounts (paper + real) in 10 minutes?

[Q] How to setup IBC in docker

[Q] Where do we get the latest versions of TWS API?

[Q] How to reset TWS connections?

[Q] which version of TWS should I use?

[Q] How to reset a paper account?

[Q] How API version maps to client/server version I see in a log?

[Q] Client can not re-connect to TWS using the same client ID after connection was not properly closed.

[Q] How to change ib primary HOST? (moving to new region)

[Q] TWS SLOW connection times?

Code Examples

[Q] I'm total noob, where to begin?

[Q] Is there a book you may recommend for a newbie like me, to really know what I am doing, what I should look for doing and what I should carefully understand and avoid?

[Q] Where can I find C++ and C# tutorials/examples using ActiveX?

[Q] Where can I find working posix C++ code examples?

[Q] Where can I find MORE ADVANCED working posix C++ examples?

[Q] Python?

[Q] Delphi?

[Q] Java?

[Q] ruby?

[Q] C#?

[Q] R

[Q] Pattern recognition library?

[Q] Reverse engineering java?

Implementation notes

[Q] How do I retrieve the bid/ask spread for given contract?

[Q] How to close an OCA group?

[Q] Order field to identify a strategy? (clientID vs orderRef vs permID)

Resolution:

[Q] Detection of loss of connectivity

[Q] Get list of all symbols by exchange?

[Q] Getting stocks / options standard deviation?

[Q] How to detect TWS disconnected from API?

[Q] What value means "no value"?

[Q] Best practice to handle ID’s / nextValidId?

[Q] After connection loss I need a list of all open orders and positions in order to get synchronized?

[Q] Tracking time (internally)

[Q] Building live Trades

[Q] Building live bars

[Q] Why do we get two identical tickSize() callbacks?

see also somewhat related:  [Q] Building live Trades

[Q] Could someone suggest a good algorithm for receiving market data for more than 100 symbols with a single account?

[Q] data links drop; feeds fade. Serious software needs to handle that. How?

[Q] Request/response/error architecture thoughts?

[Q] Thoughts on “duplicate requestID” value (bad value given by API)

[Q] Thoughts on “requestID space partitioned”

[Q] Trying to modify an order and sometimes FAIL!?

[Q] Trying to modify trailing stop, got an error 10067

[Q] Order execution messages considerations

[Q] Checking API connection / checking the timeliness of ticks

[Q] AvgPrice rounded to 2 digits, I want 3 digits!-)

[Q] How to get avgFillPrice for multiple filled order status updates?

[Q] Rounding to the tick size of the Financial Instrument

[Q] Should I use MKT or LIMIT orders? How do I track partial fills?

[Q] Why PosixMQ is a good idea for IPC? Why can’t I just use some thread-safe queue (with mutex)?

[some feedback from 100% complete frameworks out there;-]

[Q] Unit testing – which one to choose?

[other chaotic implementation notes]

Tick momentum indicator

[good implementation note on “designing stop-limit orders”]

Notes on ContractTracker implementation

Well known errors and how to avoid them

[Q] List of documented Error Codes?

[Q] reqSecDefOptParms() with exchange specified returns no data?

[Q] I observe gaps in the most recent historical date (options in my case). But later same data looks good. Why?

[Q] Errorcode(110): The price does not conform to the minimum price variation for this contract

[Q] Errorcode(162): Error message:No market data permissions (for some tickers, even with proper subscription!)

[Q] Errorcode(162) Historical Market Data Service error message:HMDS query returned no data: GCQ9@NYMEX Trades

[Q] Errorcode(162) 'Historical Market Data Service error message:Starting time must occur before ending time'?

[Q] Errorcode(162) 'Historical Market Data Service error message: Scanner filter marketCapAbove is disabled'?

[Q] Errorcode(200) No security definition has been found for the request

[Q] Errorcode(201): Order rejected – reason:It does not comply with IB's order handling rules for derivatives that result in physical delivery.

[Q] Errorcode(300): Can't find EId with tickerId:40000001.

[Q] Errorcode(321): Missing exchange for security type FUT

[Q] Errorcode(321): Error validating request:-'rc' : cause – You must specify an account.

[Q] ErrorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – Historical data queries on this contract requesting any data earlier than 199 year(s) back from now which is 18210405 00:18:14 GMT are rejected.  Your query would have run from 5240807 23:29:46 UTC to 5240807 23:33:06 UTC.'

[Q] errorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – End date not supported with live updates'

[Q] errorCode: '322', errorMsg: 'Error processing request.Only 50 simultaneous API historical data requests allowed.'

[Q] errorCode: '456', errorMsg: 'Max number of real time requests has been reached.'

[Q] Error 478 (undocumented): Parameters in request conflicts with contract parameters received by contract id: requested strike 1300.0, in contract 13.0;

[Q] Error 481, Order size was reduced

[Q] (undocumented, seen on may 2005) Error 505

[Q] Errorcode(???) Order Canceled – reason:Order size exceeds amount allowed by fat-finger check

[Q]  (undocumented, seen on Sep 2017, Apr 2020) Error 10187

[Q] Errorcode(10197): No market data during competing live session

[Q] Errorcode(10225): Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately

[Q] Errorcode(1101, 1102): Error validating request:-'rc' : cause – You must specify an account.

System Message Codes

[Q] Error: 2109 Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed.

[Q] error 2137 The closing order quantity is greater than your current position

[Q] Order size is too large IB limit is 35000

[Q] First order after IB GW API connection causes disconnect and order is ignored

[Q] Historical Market Data Service error message:No historical market data for XAUUSD/CMDTY@CMDTYBBO Last 60

[Q] While trying to modify submitted order getting: "Unable to modify this order as it is still being processed."

ContractDetails

[Q] What is the difference between contractDetails.liquidHours vs contractDetails.tradingHours?

[Q] Exchange list in ContractDetails?

[Q] Change of Timezone specs in contract details?

[Q] Incorrect Mintick?

Indexes

[Q] Is there a way to get a list of all indices available on IB? (aka "Product Listings")

Placing Orders (common)

[Q] What order types are supported by IB?

[Q] I sent an order, what should I expect next?

[Q]  How can I trade stocks during the pre-market or post market?

[Q] execDetails vs orderStatus

[Q] How do I query the status of my orders?

[Q] Retrieving past trades in an account

[Q] How to distinguish Client ids from API vs non-API?

[Q] Where to Set GTC in placeOrder API?

[Q] "update order" VS "cancel and creating order"?

[Q] How often can I modify an order?

Cancel/Modify Orders

[Q] This account cannot have open orders(including Combo) on both sides of the same US Option contract in related accounts.

[Q] How to change the limit price of an order without changing order size after partial fills?

[Q] I’m getting multiple filled orderstatus updates

[Q] Ok! Order executed! But what was the fill price?

[Q] Order Id mismatch between API and TWS

[Q] Why my order has “Inactive Order” Status?

[Q] How to cancel all orders?

[Q] How to cancel all open orders and close all positions?

[Q] How to pause order (like in TWS)?

[Q] How to get order history?

[Q] How to track ALL orders (including cancelled orders, and including manually created via TWS when my program was not running)?

[Q] I got multiple execution messages.

[Q] How to submit a sell order as soon as a buy order is executed

[Q] How do I know if my stop order is triggered from API message?

[Q] Should I use IB’s trailing stop and other advanced order types?

[Q] OCA orders?

[Q] OCA orders are simulated orders?

[Q] MOC (Market on Close) orders and OCA Group

[Q] Code: 201 – Order rejected – reason:The OCA group order has already been filled

[Q] Order overfilled?

[Q] SHORT selling (SSHORT vs SELL)?

[Q] Changing Order Quantities while order partly filled.

[Q] How do I get commissions from API?

[Q] Also if I want to see the sum of Commissions in a given day from API ?

[Q] How to check the date when the current position was acquired?

[Q] Can I use just conId for specifying the contract?

[Q] My order’s place in line!?

[Q] Max usage of buying power for DT system

[Q] openOrder / orderStatus / execDetails sequence?

[Q] Orders pacing?

[Q] How to modify the submitted order?

[Q] error: The price does not conform to the minimum price variation for this contract

[Q] Sometimes reqOpenOrders will not return all open orders. wtf?!

[Q] I have not able to retrieve the previous filled orders through the reqExecutions call

[Q] How to submit a vwap order with start and stop times and other options?

[Q] Relative Order transmit problem

[Q] How to send post-only order by api?

[Q] How to suppress: Price Management Algo Popup

[Q] Where do I get the information about how much a order was leveraged? I cannot find a property (for example, leverage) within the "Order" or "Execution" class.

[Q] How can I know when a stop limit order is triggered?

[Q] Order Limitations

[Q] How place orders before AND after market hours?

Combination Orders (aka combo order)

[Q] What are ratios for legs?

[Q] How would I set the Order::lmtPrice field for a combination order?

[Q] Please explain how to handle combo orders?

[Q] Understanding Guaranteed vs. Non-guaranteed Combination Orders

[Q] How to create option combo leg without knowing the conId of each leg?

[Q] Non-Guaranteed Combination Orders

Bracket orders

[Q] What is "Bracket Order"?

[Q] Note on the Bracket Order child order allowed types?

[Q] Bracket Order Quantity is capped? (yes – by parent's qty)

[Q] Margin for generic orders basket, multiple leg

[Q] Bracket order: Why Sell and Stop don’t cancel each other?

[Q] Bracket orders: 3 or more childs? grand child? stop order as an entry?

[Q] Bracket orders: edge cases

[Q] Bracket orders: How to place bracket order via API (recap)

[Q] How to modify pricing on a bracket order?

Placing Orders (stocks)

[Q] How can I keep a reference to these manual trades when they pop up in openOrders if all the IDs are 0 at the time?

[Q] What "Order Inactive" really means?

[Q] How to send a limit order then if the order is unfilled after a certain amount of time have the order be cancelled and do market order?

[Q] order.m_account = null ?!

[Q] m_shortSaleSlot's field?

Placing Orders (options)

[Q] What is the IV in options model used by IB?

[Q] How to get Option BID/ASK and GREEKS after the market close?

[Q] How to get daily volume for options?

[Q] See also: How to request contract details in order to get the options for a specific future?

[Q] How to Monitor Stock Loan Availability?

[Q] How to find SLB info?

[Q] TWS and Option Spreads

[Q] weird theta

[Q] How should an API client detect an option exercise, the exercise being done manually in TWS?

[Q] How to find out available strike prices for options ?

[Q] How to get all valid options (scrape option chains)?

[Q] How to pull the entire option chain?

[Q] Why option chain data is delayed (sometimes up to 1 minute)?

[Q] Is IB’s model reliable or useless and I need to do all the calculations myself?

[Q] Am I required to pay the dividends on shares borrowed when short selling?

[Q] Which stocks are shortable which are not?

[Q] Unable to borrow?

[Q] How to determine the pricing step for an option?

[Q] placeOrder() No security definition has been found for the request

[Q] I see duplicate events from time to time!

[Q] attached order

[Q] Bad equity options spread (bid-ask seems to widen significantly)

Placing orders (futures)

[Q] Can I use SMART for futures?

[Q] How to request contract details in order to get the options for a specific future?

[Q] Futures – No security definition has been found

[Q] Can I place MOC order for es future?

[Q] Futures options data via API

[Q] How to get Futures Front Month

[Q] continuous contracts rollover schedule

[Q] Continuous contract – how to determine offset?

[Q] How to create a true continuous contract historic data (futures)? Do you also perform back propagation of price adjustments when you roll?

[Q] Futures: what is “First Position Date”?

Combo Orders

[Q] Placing Combo orders?

[Q] Combo order example:

[Q] Get info on all legs of placed combo order?

(quote)

[Q] Note: good-after-time (GAT) orders are not supported for ("generic") combo orders!

[Q] Can place orders for each leg, but can’t place combo!? wtf?

[Q] Is there a way to get contract details of the BAG contract itself?

[Q] Combo example please!

[Q] IB Lag with Combination Orders

[Q] Can’t change qty on combo order!

After trade is complete

[Q] How to get all Completed Trades within the last N days?

[Q] tell me more about Flex Query

Paper vs real accounts

[Q] Accuracy of paper-trading account

[Q] PaperTrading – how do they simulate fills on limit orders?

[Q] Paper account vs. Demo account?

[Q] Partial fillings of MKT order on paper trading account? wtf?

Limitations of the PaperTrader

[Q] I am wondering if there is any way to tell from the API whether the socket is connected to a paper trading account or a real account?

[Q] Trying to place pegged to stock orders through API or TWS via paper account?

Charting

[Q] Any easy way to do charting?

Fundamental data

[Q] How to get SPY dividends?

[Q] reqFundamentalData() – How to Parse XML File?

[Q] How often Fundamental data updated?

[Q] Why only 4 out of 6 fundamental requests succeed and for 2 of 6 I got errors:

[Q] You mentioned "we no longer directly provide support for API Ver. 9.67". How does this work? Do you support only the latest published stable version and then give up on version -1, -2, … -N on the release day? What are the rules here?

IB Data Subscription

[Q] How IB throttles / samples it's level 1 data? (aka "best description of

what's going on with the IB feed")

[Q] Is HistoricalData volume represented in the hundreds?

[Q] NYSE auction order imbalance data?

[Q] How to download broad market statistics, such as NYSE  TRIN, NYSE Advancing issues, NASDAQ ?

[Q] tickPrice vs tickString?

[Q] Would I get the same data if I subscribed to the same instrument from a paper and real account? Would it count as "one line"?

[Q] Can I get historical price data of instruments which have expired?

[Q] Trust daily close or 1-minute data close?

[Q] VWAP through the API

[Q] What RTVolume fields are?

[Q] Realtime Volume timestamp sometimes goes backwards?

[Q] Is there an easy way to get the S&P 500 index component stocks?

[Q] Can I download the historical data of obsolete symbols?

[Q] My accumulated volume does not match the TWS one! Wtf with VOLUME?

[Q] Does it make sense to use separate data / broker connections for separate tasks?

[Q] API and Trading hours?

[Q] How to get the day open price for NYSE stocks?

[Q] How to get Futures Open Interest?

Data subscription: Volume

[Q] How to get somewhat accurate cumulative volume?

Data subscription: Dividents

[Q] Get Dividend information?

[Q] Dividend Calendar (dec, rec, div, exdiv, amount, etc)

[Q] Dividend Adjustments? Adjusted Previous Close not available on IB?

[Q] API reqFundamentals dose not return previous dividend data for index

[Q] Data subscription: Calendar events

[Q] I got invalid update of bid/ask which leads to CROSSED PRICE. wtf?

[Q] Crossed quotes on IB (again)

[Q] Snapshot market data vs “real time” data.

[Q] Market Data Stuck when requesting snapshots!

[Q] reqRealTimeBars vs reqMarketData – how to get real time bars?

[Q] How to get list of all IB’s tradable contracts?

[Q] Sometimes lastSize=0. wtf?

[Q] When is the exact start of a new bar? Will hand-made bars (from tick data) be exact same bars as I receive from IB?

[Q] Why IB uses sampled data instead of plain real-time tick streams? Aka "fudge" factor for tickSize event.

[Q] What are the differences between Level I and Level II data ?

[Q] Which request ID should I use?

[Q] How deep can I go in history?

[Q] Bug in historicals? Price but no volume and other historical “holes”!

[Q] Options historical data?

[Q] How to store historical data?

[Q] reqHistoricalData: Volume vs Count?

[Q] How to backtest?

[Q] high, low, close fields in tickPrice ?

[Q] What is the sequence of tickSize() tickPrice events?

[Q] I get negative and zero quotes! wtf?

[Q] How to get all strike prices for a given ticker & expiry?

[Q] No security definition has been found.

[Q] reqMarketData(): For most stocks I get the last prices, however for some stocks I only get bid.

[Q] reqMarketData(): sometimes getting back tickPrice events with a price of -1

[Q] What is conId?

[Q] Volume looks incorrect?

[Q] How to get indices historical prices?

[Q] How to get realtime P&L?

[Q] reqRealTimeBars() – "whatToShow" string?

[Q] reqRealTimeBars() – only 5sec bars available? Partial bars?

[Q] tickPrice(): canAutoExecute field – wtf?

[Q] When requesting historical data I often get “Invalid step”

[Q] Valid Duration and Bar Size Settings for Historical Data Requests

[Q] Is it possible to develop TWS like depth book using reqMktDepth() call?

[Q] implementation notes: reqMarketDepth

[Q] What happens when I do reqMktData() afterhours?

[Q] IB's own historical data corrections (history of the history:)

[Q] can reqTickByTickData get delayed ticks?

TWS Scanners

[Q] Where do I begin with scanners?

[Q] Biggest gainers since open between $5 and $10?

[Q] Market Scanner Pre/Post Market Hours

[Q] Trouble getting API scanners to output the same date as the one in TWS!

[Q] Give me simple example of working scanner request.

[Q] Can't get scannerSubscriptionOptions to work

[Q] Note on contracts which may not be accessible through TWS when running scanners:

[Q] Market Scanner Pre/Post Market Hours:

[Q] How to find the location code for Australian when creating a scanner subscription object?

IB News subscription

[Q] When subscribing to a news feed via API, is it possible to receive the rank?  If yes, how?

Basics for beginners

[Q] Vocabulary

[Q] Limitations on quantity of symbols?

[Q] Where can I find the API version info?

[Q] I’m about to post a question to IB API list. What should I do?

[Q] Number of hops to IB?

[Q] Is IB real time data actually “real time data”?

[Q] When I pass a BUY LIMIT order on the ES, let's say at 1300.25, I always have to wait until the price goes to 1300 to get a fill…is that normal?

[Q] Will I have this problem if I try my algorithm with a stock like IBM on SMART?

[Q] I am behind a firewall I like to know the TWS out bound ports?

[Q] Other TWS API FAQs?

[Q] Any general recommendations?

[Q] IB commission rate comparison

[Q] How to sync market data?

[Q] Notice of High Order Ratio

[Q] Some IB API timing requirements

Historical Data Limitations

[Q] What is Pacing Violations?

[Q] Can I be long and short on the same underlying same time?

[Q] Any historical-data aggregating programs?

[Q] How to diagnose an issue?

[Q] Why should NOT I use DDE?

[Q] Is demo account API behavior differ from the behavior of real account?

[Q] What are "EX" methods?

[Q] custom indicators?

[Q] Any other tips/hints for beginners please?

IB Gateway app related

[Q] Installing IB Gateway and TWS API for linux?

[Q] Trader Workstation (TWS) vs IB Gateway?

[Q] Any performance difference between IB Gateway vs TWS app APIs?

[Q] Gateway Crash on Startup – Linux

[Q] list of the error 321 "message is not supported"

[Q] autologin script for IB Gateway?

[Q] How to automatically accept client connections to TWS and/or Gateway from any remote IP address (not known in advance)?

Some Forex notes

[Q] FXCONV support in API?

[Q] IDEALPRO LMT orders at the top of the order book never executed

[Q] Please check my Forex math!

[Q] according to the documentation m_avgPrice should include commissions, but it doesn’t!!

[Q] Can't get some Forex historical data?

[Q] Why I don’t get TRADES from “real-time” FX ticker?

[Q] Tracking Forex positions?

[Q] IDEALPRO's hours?

[Q] Minimum size requirements for IDEALPRO

[Q] Minimum Price Increment

Glossary

Other

[Q] Using Managed Accounts: How to set deposit size on subaccount / disable loan?

[Q] Best take profit order type

[Q] contract->exchange = "SMART" works, but contract->exchange = "NASDAQ" does not!

[Q] IB technical support contact info?

[Q] Fully clarifying exchange vs primaryExchange

[Q] I need some actively trading stock for my midnight debugging

[Q] Where do I get the latest standalone TWS?

[Q] How do I know if the market is open via API?

[Q] How to get account balance, transaction fees, cash transactions, Net Asset Value, etc.? (Flex?)

[Q] How to automate Flex Reports?

[Q] Which Java is better to use for TWS on Ubuntu?

[Q] Is there any way to see S&P 500 index on TWS?

[Q] Release notes – should I read it?

[Q] French transaction tax, how to anticipate?

[Q] Getting exchange location for order?

[Q] What is the "numIds" for in reqIds function?

[Q] Does anybody know of a free technical analysis libraries?

[Q] Is there a way to pull these daily libor rates directly from TWS?

[Q] Closing Prices in TWS Are Wrong?

[Q] How to place MOO orders?

[Q] How to update the future's margin requirement from TWS API?

[Q] How to get margin requirements for possible trade?

[Q] [TWS API] Stress-testing margin requirements using Risk Navigator to model changes in the underlying price?

[Q] Figuring out Lookahead / Overnight Margin Rate For A Stock

[Q] Index of all available addons that can be purchased for tws?

[Q] Reverse engineer IB API socket protocol

[Q] Read this later for TAX purposes (if you ever need it at all)

[Q] Can I get the MarketValue for all the currencies?

[Q] Managed Accounts?

[Q] Getting a full list of Exchange Listings via API?

Reporting

[Q] How Do I Match the OrdeId that is sent with the Executed Order ID that appears in the Trades and Summary tabs?

[Q] orderStatus function is be called 3 or 4 times(!) even when there is no update of status. Wtf?!

[Q] How to track order fills?

[Q] How to obtain the complete current portfolio?

[Q] Is there a way I can get total short value of my account via the API? I'm basically looking for the total value of all short positions.

[Q] Reported realizedPNL got reset in the random time during the trading hours, how to change that time?

[Q] How to get PnL of Trades?

Not yet confirmed (but possible issues)

[Q] Validity of Paper Account for Testing Algo Trading Outcomes?

[Q] Other limits I’d be aware of?

[Q] there is no correlation between bid/ask and trades in IB's feed (?!!)

[Q] updatePortfolioEx() non consistent behavior?

[Q] Access multiple accounts?

[Q] How to monitor events (keyboard, mouse) in X?

APPENDIX

[Q] Good alternatives to IB?

Time drift

[Q] Does my/your/any computer have a time drift?

[Q] Should I build my own NTP server (on my LAN) or would it be sufficient to just use public ones?

[Q] How to build own Stratum-1 NTP server on Raspberry Pi?

Some unsorted yet related stuff


About this FAQ

Purpose

Let us learn and make TWS API better and easier to use/understand.

This document is Dmitry’s personal attempt to learn TWS API and 99.9% of its content came from TWS API Group https://groups.io/g/twsapi. I tried to structurize some questions and answers in form of FAQ grouped by topics.

Scope

 I tried to collect all the general Q/A’s about data flow, tricks of certain functions of TWS API, limitations, workarounds and other general-level info. On the lower-level side of things (language specific) I collected mostly C++ / Java related topics and did not store much on topic of Windows Sockets, ActiveX, CSharp, VB or DDE for Excel etc. Want to add missing topics? You’re welcome to participate and add it yourself!  🙂

In Dec-2014 I started digging (reading through) all the history starting from messages around 05-Feb-2010 and this FAQ is an ongoing attempt to keep track on all important topics.

Disclaimer

This FAQ represents a small fraction of the TWS API Group discussions and by no means is intended to replace one. Please search online group archives for many more answers (than you can find here).


Link to the latest IB API documentation

[A] by Josh

On Wed, May 3, 2017:

The newer documentation is at: http://interactivebrokers.github.io/tws-api/ 

by the way.

Josh

–also–

———- Forwarded message ———-

Date: Thu, 18 May 2017 17:29:38

From: jbeacomib <notifications@github.com>

Reply-To: InteractiveBrokers/tws-api

To: InteractiveBrokers/tws-api <tws-api@noreply.github.com>

Subject: gh tws-api]  Stale IB doco relative to the github  content (#493)

The documentation which is actively updated with the most

current information about the builds of TWS and API versions

that are on the website (API versions 9.72, 9.73) is hosted on

[Github](http://interactivebrokers.github.io/tws-api/)

The older documentation was created for version 9.71 of the

API and previous builds of TWS. It is now no longer updated

and has been superseded by the Github documentation site. The

new documentation is correct that up to 32 API applications

can now be connected to one instance of TWS or IB Gateway.

However in practice, this limit is uncommonly reached as it is

usually more efficient to reduce overhead with a smaller

number of connected clients.


TWS app related

[Q] How to turn off Security Code for TWS real account login?

[A] View/Reply Online (#768)

Added on 09-Jul-2020

There is a setting for this at online Account Management. Follow this:

Online account → Settings → User Settings → Secure Login System → Secure Login Device Opt Out → tick the box at “I only want to use my Secure Login Device when logging into Account Management”.

Then press "continue". You will be presented with some disclaimer forms which you have to acknowledge.

If you later change your mind you are able to enable 2FA again, using the same procedure.

I hope this helps.

[Q] How many market data lines one has subscribed to in TWS?

[A] View/Reply Online (#43856)

Added on 23-May-2020

Press Ctrl-Alt-= simultaneously on your keyboard, TWS will show pop-up window with number of lines used.

See also:

https://ibkr.info/article/3292

[Q] How to import Tickers from a File?

[A] discussed here (View/Reply Online (#43800))

https://www.interactivebrokers.com/en/software/tws/usersguidebook/getstarted/import_tickers_from_a_file.htm  Here's copy-pasted article from IB's website in case the link above no longer works:

Import Tickers from a File

You can populate a trading page with market data that you import from a comma-delimited .csv file or .txt file. Create the list using any word processing program and save the file with a .txt file extension, or in MS Excel and save as a .csv file.

To create file of ticker symbols to import

  1. Create a file with the extension .txt, for example twsTickers.txt, or create a .csv file in Excel.
  2. In the text file, create one comma-delimited line for each line of market data you want using the syntax described below. In Excel, create one row, using one cell for each field. Note that the first field designation is the line type: DES, SYM, and CON.

DES, UNDERLYING, SECTYPE, EXCHANGE, LASTTRADINGDAY, STRIKE, PUT/CALL, MULTIPLIER

SYM, SYMBOL, EXCHANGE

CON, CONID, EXCHANGE 

The DES line type requires information or placeholders for information, with the strike, Put/Call and multiplier fields being optional. Format for the Last Trading Day is YYYYMM, or YYYYMMDD. All entries must be in caps. An example for an options contract XYZ would be:

DES, XYZ, OPT, ISE, 201509, 75, CALL,

A stock instrument for symbol XYZ in this line type would look like this:

DES, XYZ, STK, SMART,,,,

where the commas are placeholders for fields you don't define for stocks.

Using the SYM line type for a stock would look like this:

SYM, XYZ, SMART

Use the CON type to quickly enter contracts with a conid, for example

CON, 12348765, SMART

To import a file into Classic TWS

  1. From the File menu, select Import/Export and then select Import Contracts.
  2.  
    Use the Browse button to find the file to import.
  3. Specify where you want imported tickers to go, either onto an existing page or onto a new page.
  4. Click OK.

To import a file into a Mosaic Watchlist

1.  Right click in the Watchlist, and select Import/Export then Import Contracts.

2.  Use the Browse button to find the file to import.

3.  Click OK.

Imported contracts entered onto an existing page appear at the bottom of the active trading page or Watchlist.

[A] by Josh

typing a local symbol directly into an watchlist row will bring it up immediately, because it is unique to that option

i.e. 'AAPL  200221C00322500' (make sure you have the correct spacing)

Option Symbology Initiative (OSI) standard: https://ibkr.info/article/972

[Q] Link to the official server reset time?

[A] by Josh

https://www.interactivebrokers.com/en/index.php?f=2225

[Q] How to check and solve connectivity issues affecting the Trader Workstation (TWS)

[A] by Josh

https://ibkb.interactivebrokers.com/article/2807

(I’ll preserve a quote from the link above just in case here;)

Background:

The Trader Workstation (TWS) software needs to connect to our gateways and market data servers in order to work properly. Connectivity issues affecting your local network or your Internet Service Provider network may negatively affect the TWS functionality. In this article we will indicate how to test your connectivity using an automated connectivity test web page.

 

How to test the connectivity using the automated "IB Connectivity Test" web page?

1) Click on this link: http://www.interactivebrokers.com/cgi-bin/conn_test.pl 

2) Please wait until all the tests have been completed and results have been displayed. If you see "Success" as outcome for all tests, your connectivity to IB Servers is reliable at the present moment. No additional connectivity troubleshooting or configuration should be needed.

3) If you see "Fail" as outcome for one or more test/s, please click on the link "Fail" itself in order to display the "IB Network Troubleshooting Guide". That section will help you conduct some manual tests to identify the cause of the failure.

Note for Corporative environments and Proxy server users: the automated "Connectivity Test" page may return misleading results in case your machine is accessing the Internet through a Proxy server. This usually happens if you are connected to a company network. If this is your case, we kindly ask you to contact your Network Administrator or your IT Team and ask them to perform a manual connectivity tests towards the destination servers indicated in the table on the top of the IB automated "Connectivity Test" web page itself. The manual connectivity test should be conducted using destination TCP ports 4000 and 4001. Should they prefer to have the server list in another format or should they need to set up the firewall / IP Management rules, you can forward them this page.

(end of quote)

[Q] How to run 2 TWS on different machines (to access same account)?

This question was posted on github ib-controller project:

Hi!

Any chance you can add a setting to deal with the "someone has logged in with this username from another location" scenario?

IB's "great" monster doesn't let you log in from two locations so I'd like to be able to login from my phone to check my portfolio at times that I know the TWS API will not be sending orders.

I would imagine it would click the button after some timeout.

Let me know what you think!

Thanks

Georgios

Added on 16-Aug-2016

[A] by Ben Alex

One option is a second username on your account. Full info at http://ibkb.interactivebrokers.com/node/1004.

[Q] How to revert installed version of API properly? aka downgrading API

Added on 29-May-2015

[A] by Richard L King (found here)

Well, I’ve discovered why I was unable to get any previous ActiveX API version working after uninstalling 9.72 Beta..

When you install the TWS API, the install process puts a library called TwsSocketClient.dll into the Windows system folder. This library has a file version number the same as the API version, and the ActiveX control (Tws.ocx) is actually a wrapper around this library, so Tws.ocx and TwsSocketClient.dll need to be the same version for the ActiveX API to be guaranteed to work properly.

The problem arises from the fact that when you uninstall the API, the TwsSocketClient.dll is not removed. If you now try to install an earlier version of the API, the existing TwsSocketClient.dll is not replaced, presumably because the version number being installed is lower than the one already there. So now the version installed does not match what the ActiveX control expects, and so it doesn’t work – at all.

So the solution is simply to manually delete TwsSocketClient.dll before attempting to install an earlier API version.

rholowczak, perhaps it would be useful including this info in your ActiveX tutorial? Most people probably never revert to a previous API version, but they may want to if, for example, they’ve been trying out a beta version and want to revert to the latest supported version.

[A2] by newguy@videotron.ca

Added on 30-May-2015

That's actually documented by IB.

Uninstalling and Re-installing the TWS API Software on Windows

Uninstalling and Re-installing the TWS API Software on Windows

Open topic with navigation You are here: Overview > Uninstalling and Re-installing on Windows Uninstalling and Re-installing the TWS API Software on Windows

View on www.interactivebrokers.com

 

[Q] How to optimize TWS Java VM performance? aka the "latest and greatest" JVM server settings

Added on 27-Apr-2015

[A] by FutureScalper (found here)

you should use IBController by Richard L King

I would emphasize that the most significant thing you can do to improve

performance is to use the Java Server VM. I have a high performance

multi-threaded Swing-based GUI applications which does realtime analysis and

order processing, etc. It runs using the Server VM, even though it is a

"client" application. I also run TWS using the Server VM. This provides a

range of tuning options and global optimization capabilities which probably

doubles or triples throughput.

You can get the Server VM only by installing the JDK, and then you may want

to move the jvm.dll into your JRE area or whatever allows you to use the

command "java -server -version" to verify its installation.

Again, running TWS itself under the Server VM as well is critically

important for high performance.

Here is an example of the command line which I use to run standalone TWS

under the Java Server VM on Windows. Various options are technical, and

subject to change.

c:\WINDOWS\system32\java.exe -server -dsa -Xbatch -Xss192k -Xms200m -Xmx200m

-Xnoclassgc -XX:+ForceTimeHighResolution -XX:-TieredCompilation

-XX:+RelaxAccessControlCheck -XX:MaxInlineSize=64000

-XX:-DontCompileHugeMethods -XX:CompileThreshold=250 -XX:CICompilerCount=1

-XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=150 -XX:+DisableExplicitGC

-XX:ThreadStackSize=192 -XX:+AggressiveOpts -cp

jts.jar;pluginsupport.jar;jcommon-1.0.0.jar;jfreechart-1.0.0.jar;jhall.jar;other.jar;riskfeed.jar;rss.jar

jclient/LoginFrame \jts

The Server VM will compile "huge" methods, and will "inline" code, and

perform global optimizationsn not possible with the standard Client VM.

Performance and scalability matter for many applications.

[Q] How to make TWS application running 24/7 automagically?

[A] you should use IBController by Richard L King

URL: https://github.com/ib-controller/ib-controller

You can download the latest version from
https://github.com/ib-controller/ib-controller/releases/latest

(quote-1)

One possibility is to use IBController, which is a Java program that loads

TWS and intercepts various window events and handles them automatically.

Note that IBController can only handle logon if you have opted out of the

security device scheme.

You can find IBController in the Files section of this forum, under Auto

Login Codes.

Richard

(end of quote-1)

(quote-2)

IBController is a Java application that was written to enable Interactive
Brokers Trader Workstation (TWS) to be run in "hands-free" mode.  This
makes writing unattended automated trading systems possible.  IBController
automates the TWS login by filling the login dialog with your login
credentials. It also handles the dialog boxes that TWS presents
during programmatic trading activies.

(end of quote-2)

link: http://sourceforge.net/projects/ibcontroller/files/

few random notes from IBController users:

—————————————————————————————————————————————-

by Mike Smith mikesmithv@gmail.com

You might try changing the MaxPerSize parameter to something larger, like from 128M to 255M.  When that number is too small it can cause any number of weird symptoms.  When there is not enough space to load a new class then a previously loaded class is unloaded and strange things can happen.  Programmers sometimes assume static class variable are "global variables" that remain as long as the application is launched and that's not the case if the class is unloaded.

Not long ago TWS used the default MaxPerSize and that was not enough if launching under IBController.  Adding -XX:MaxPermSize=128M the the IBController launch command was needed because of the extra classes loaded by IBController.  Now I see on IB's web site the launch command for TWS now contains XX:MaxPermSize=128M just for a normal TWS launch.  So it appears they have added more classes to TWS so they had to bump up that number for a standard launch.  So it makes sense a larger number is needed for launching with IBController now.

I don't use IBController anymore so all this is based on my past experiences with it.  Now I use the gateway restarting it once a week and just let it run all week on one log in.  So all this "in theory" and not my direct experience.  I hope it makes sense.

see also discussion:

Optimal memory setting

https://groups.io/g/twsapi/topic/4047139

—————————————————————————————————————————————-

[Q] How to set up IBController (3.2.0) + TWS (build 956)  on headless Ubuntu 16.04 LTS to run TWO accounts (paper + real) in 10 minutes?

[A] https://dimon.ca/how-to-setup-ibcontroller-and-tws-on-headless-ubuntu-to-run-two-accounts/

Added on 13-Jun-2016

Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.

[Q] How to setup IBC in docker

[A] by dvasdekis <notifications@github.com>  view it on GitHub

Added on 19-Jul-2020

I maintain an updated docker container with IB Gateway v978.2c and IBC v3.8.2 here. Feel free to link to it instead of maintaining an independent solution.

[Q] Where do we get the latest versions of TWS API?

[A] http://interactivebrokers.github.io/

[Q] How to reset TWS connections?

[A]

Ctrl-Alt-R resets the account server connection

Ctrl-Alt-F resets the market data connections

[Q] which version of TWS should I use?

[A] by Ruediger Meier sweet_f_a@gmx.de

Actually I install all TWS versions in parallel just to be able to try

it out whenever I will be in the right mood.

The default one which I also use for production is the stable version

936.9l.

For those who don't know it, IB provides stable and latest versions. The

current stable TWS gets only bugfix updates but no new features. That's

why it has such a high minor version number (letter).

[Q] How to reset a paper account?

[A1] View/Reply Online (#46218)

Added on 25-Jan-2021

This, from IB, worked:

You can reset your paper account equity to a value different than the original and up to five times your production account value. Please note that reset requests should be entered before 16:00 EST in order to take effect for the next business day.  

Please follow below steps to reset paper equity balance:  

1. Log into your paper trading Client Portal by clicking the toggle on the login screen  

2. Select Settings followed by Account Settings  

3. Click the Configure (gear) icon next to Paper Trading Account Reset  

4. Select the reset amount from the drop down menu provided and click Continue.

[A2 (oder – from around "before 2017")] by Eric J. Holtman

Under "Trading Access", pick "Paper Trading Account Reset".

— or —

Log into paper account management (don't log into regular account)

https://www.interactivebrokers.com/sso/Login

Left side of screen – "Trading Access" – under that – "Paper Trading

Account Reset"

It doesn't kick in until next trading day.

by Jim Covington

[Q] How API version maps to client/server version I see in a log?

[A] by IB support (by phone)

Client side:

API 9.67 —> Client version 59              TwsApiC++.9.67_3.zip

API 9.70 Release Date: October 2013 —> Client version 62

API 9.71 Release Date: Aug 6 2014 —> Client version 63 

Server side:

build 946 —maps-to—> TWS socket server version 71

[Q] Client can not re-connect to TWS using the same client ID after connection was not properly closed.

[A] by Dmitry

https://groups.io/g/twsapi/topic/4047690

And followed discussion in the hijacked thread 😉

Same topic was also discussed here:

https://groups.io/g/twsapi/message/37644

Basically it is about: when client closes connection to TWS abruptly then client often can not reconnect using same client id. One would need to restart TWS or wait for things to self-release on TWS app side (up to about 1 hour).

[Q] How to change ib primary HOST? (moving to new region)

Just got a box in NY4 but my IB account still connects to ASIA – hdc1.ibllc.com even though I choose region as – America at login window…

Any config I can change to force IB connects to USA East – ndc1.ibllc.com?

[A] by Evgeni Andreyev e.andreyev07@gmail.com View/Reply Online (#45559) 

Added on 03-Oct-2020

No, you have to explicitly request (from IB Support) moving the IB server assigned to your account at registration to a new location. This will  then happen over the weekend.

[Q] TWS SLOW connection times?

[A] by JR <TwsApiOnGroupsIo@reinold.org> View/Reply Online (#46681)

Added on 03-Mar-2021

Since the tally of folks that have the same issue seems to be approximately zero, IB having an issue feels unlikely, though not impossible. So let me briefly raise three things that I'd look at if I'd have to troubleshoot API connection issues.

[1] Stale API connections:

We once wrote a simple "Is my TWS alive and actually connected to IB" monitor that would connect to TWS, do a simple operation that requires TWS to communicate with IB, and then disconnect. The monitor was started automatically and frequently but somehow caused stale (not completely closed) API connections to build up. When you click "DATA" to get the connections listing in TWS, do you see stale API connections from past script executions? Those did cause connection request delays, in my case, even when TWS was far away from the maximum number of API connections.

[2] DNS reverse lookup:

You said you checked the network on your server, but did you check DNS as well?

  • Have you experienced any sluggishness (even briefly) for regular commands on your server? Assuming it's a Linux system, even commands such as "ls -l"
  • Does "netstat -a" or "lsof" hang for a few seconds but "netstat -an" or "lsof -n" do not?

Several library functions unexpectedly perform "reverse DNS lookups" and can block the caller (in this case the thread in the Java JVM that handles your incoming API request to TWS) for extended periods of time until failing DNS lookups succeed or time out. If your internet service provider has any instability with reverse DNS lookups, otherwise perfectly fine servers can have operations seemingly randomly freeze for 20 to 30 seconds.

[3] How are your entropy levels?

This is a little stretch but fits the symptoms you are having if you are running some kind of Linux system. And it took me weeks to figure that out on a server I had in a data center in Dallas (though this is not specific to Dallas or Texas)

In order for strong SSL encryption to work, the Linux kernel collects "truly random events" and builds up a buffer of entropy that Java JVMs (and other programs such as sshd, httpd, …) access through the /dev/urandom device. And while only very few bytes are needed each time, opening or reading from /dev/urandom when the buffer is low or empty will block the caller until enough entropy has been collected. And the Java JVM frequently opens /dev/urandom when various network related operations take place.

You can read up on this with "man urandom" and check your server's entropy level with "cat /proc/sys/kernel/random/entropy_avail"

Hope this helps.

Code Examples

[Q] I'm total noob, where to begin?

[A] by ExStock View/Reply Online (#45602) 

Added on 10-Oct-2020

I also started from scratch, so I don't mean to discourage you, because it can be done!  But there is a lot that you'll probably need to learn in order to do what you want safely.  (Be sure to spend a lot of time working with an IB paper trading account before trying anything out for real!)

One place you might want to start are the videos about the Python TWS API available from IB: https://tradersacademy.online/category/trading-courses/ibkr-api .  These videos are a bit like drinking from a firehose–lots of information very fast–but they do cover a lot of the basics, and you can just hit pause a lot.  

With those videos as background knowledge (and they do show the actual Python code involved for each example) you might want to consider installing the ib_insync Python framework, which simplifies the process: https://ib-insync.readthedocs.io/index.html .  If you click on the Notebooks link, you can see more code examples.

You might also want to install either Jupyter Notebook or JupyterLab (you don't need both): https://jupyter.org/ .  They both make coding in Python simpler, as well, and will make what you see in the ib_insync notebooks easier to follow and experiment with.

Finally, if you do get into ib_insync, there's also a groups.io for them: https://groups.io/g/insync .

All of the above is free to play around with–good luck!

[Q] Is there a book you may recommend for a newbie like me, to really know what I am doing, what I should look for doing and what I should carefully understand and avoid?

[A] View/Reply Online (#45571)

Added on 06-Oct-2020

Algorithmic Trading with Interactive Brokers by Matthew Scarpino has some basic python scripts and a few list of parameters that is useful for a beginner.

That book is good, I'd also recommend looking at Rob Carver's blog (https://qoppac.blogspot.com/) he's got a section (March 2017) on how to use the IB API, it covers the basic stuff connecting, getting prices, trading, reconciling which you can then build on.

[Q] Where can I find C++ and C# tutorials/examples using ActiveX?

[A] by Rich Holowczak (a.k.a. “Prof. H.”) (src)

Folks

A while back a wrote a series of tutorials for the Interactive Brokers API for my Financial IT classes. I am now getting around to assembling them on my web site.  I have a few going up now using Visual C++ and Windows Forms. I will be posting more over the next few weeks that include WinForms with C# as well. Your comments and suggestions are welcome:

http://holowczak.com/category/ib/

Cheers,

Rich H.

[Q] Where can I find working posix C++ code examples?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A1] by Jim Covington

Sample code is on your machine inside the IB API installation folder.

[Q] Where can I find MORE ADVANCED working posix C++ examples?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] There is a great IB API Wrapper TwsApiC++ by Jan Boonen (see on GitHub)

(quote)

In the C++ implementation, the basic two API classes are EWrapper and EClient, in combination with the other classes/structures defined in the directory 'Shared'. Comparing with Java, these classes are pure interfaces. The implementation of EClient is done in  EClientSocketBaseImpl and at linking time, all is connected. The implementation of EWrapper is up to you.

Almost every C++ programmer opts to write the definition of a class and its implementation in two distinct files, although C++ does not enforce it. I prefer more the Java way, although C++ does not support it to the same extend. Some care is needed, and when pure interfaces are needed, the definition and implementation has to be separated anyway. So when studying different C++ projects, you will discover different approaches. Discussing all the pro's and contra's is a never ending story, as are the programming text style discussions.

The non-posix version of the api cannot be used as stand alone. It depends on the Windows specific socket implementations for reading the socket and calling the EWrapper methods. So the 'EReader' is somewhere down deep in the windows specific asynchronous socket stack.

This makes the non posix version non-portable and unusable for a non-windows program. To overcome these two weaknesses, some free libraries exist. I started the free TwsApiC++ library in 2008. The current version is based on the posix library of IB.

What does this TwsApiC++ library offers you:

a) the EClient and EWrapper are still the basic two API classes (or more specific, a derived version of it EClientL0 and EWrapperL0). So no need to access all the internal classes such as EClientSocketBase, or the other complex posix classes.

b) it implements an EReader, which is turned on by default. It runs in a separate thread, and waits for the incoming data on the socket, and calls the EWrapper methods without any delay. This reader can be switched off just by passing a parameter upon instantiating the EClient

c) it implements an empty method for every EWrapper method defined. So you only have to implement the one you are interested in

d) It protects the inner working of the IB library, where the TwsApiC++ is a wrapper around, from the possible exceptions thrown in the users code, but not catched by it.

e) It protects the EClientSocketBaseImpl from concurrent use of its internal members, which could lead to loss of requests send to the TWS

f) It ensures the data stuck in the buffer of EClientSocketBaseImpl  and failed to be send to TWS for some reason, is send as soon as possible, and not sits there waiting until the next EClient call is made. In the posix api, you the programmer has to take care of that(!)

g) optionally, it provides a way to check the correct spelling of the many textual or numeric parameters upon compile time instead of runtime.

The library is small and fast.  A few examples/test programs are included, and also project files for Visual Studio and make files for linux/unix are included

It includes a readme file.

See it on GitHub:

https://github.com/JanBoonen/TwsApiCpp

Or for group members:

See Files  > C++ Code > TwsApiC++ Directory

There are other implementations too, other members of this group know more about these.

Have fun with it.

Jan

(end of quote)

see also tutorial how to install TwsApiC++ here: [Q] Installing IB Gateway and TWS API for linux? (tutorial by Rich Holowczak (a.k.a. “Prof. H.”))

Note from Kurt:

when I *was* using TwsApiC++ I had to do extra work to *undo* the fact

that (like Java, I guess) TwsApiC++ creates a separate thread to read the

socket, and callback messages come in on that "reader thread". So I needed

another layer of buffering to get the callbacks to come in on the main

thread, and this required a tricky set of function overloads which pretty

much required modifying the TwsApiC++ implementation classes due to how

things were (not) factored.

The posix TWSAPI either does not require any such effort. Either there is

not a separate reader thread, or the buffering is already taken care of. (I

can't remember which.)

-Kurt

Another note from Jan came from here

JanB

Message 16 of 28 , Oct 20, 2010

View Source

Hi all,

I second this: the new improved API shouldn't reinvent the wheel, but rather make the current IB API a 'more round wheel and easier to turn' (hope this makes sense).

As Shane Cusson states, implementing a simple strategy with IB's API is 'like using a bazooka to swat flies'. So this new API should simplify its use, but should not prevent the implementation of complex strategies either.

For the C++ implementation, something like TwsApiC++ is a first step as it hides the use of the sockets. Not ESocketClient is the interface, but EClient/EWrapper is. Having to deal with the sockets in your ATS is fundamentally wrong, isn't it? TwsApiC++ more or less levels the C++ API with something like the API from dinosaurtech for C#.

On top of that, I'm implemented/ing two extra layers and I thought it could be interesting to share them here:

In a first layer on top of EClient/EWrapper (EClientL0/EWrapperL0):

1) The functionality of EClient/EWrapper is regrouped in functional classes such as Connection, MktData, HistoricalData, PlaceOrder, Account, Portfolio, etc. Each class keeps the data needed to make the request, and all EWrapper events related to the request are routed back to the requesting class. I.e. each reqHistoricalData to be executed needs its own instance of the class HistoricalData, which handles the request and receives all the historicalData() events for only his request.

2) It hides all the Id's (tickerId, orderId, reqId ): the allocation of the id's is automated and hidden. Errors with id==-1 are routed to the Connection class.

3) The classes for Account and Portfolio filters the events on the acctCode, so each account can have its own instance of Account and Portfolio. A blank/empty acctCode makes it to receive the data for all the accounts in case of.

4) Each of these classes have a common interface of a Request(), Cancel(), Resend() and Error(). The parameters for the req… are set during the allocation of the classes, so the Request() method has no parameters.

As the first layer resembles the EWrapper interface and does not adapt to any special behaviour of the IB API, the second layer goes one step further to make live easier for the programmer as follows:

1) it stores the data for classes such as Account, Portfolio, HistoricalData etc. (not for Mktdata or RealTimeData i.e. )

2) offers a more uniform interface when possible: OnInit() (called after the …End() events meaning all Account is available, Historicaldata is complete etc), OnChange()(i.e. after the Init() was called, and new updated Account data receives), and a ForEach(), forward and backward Iterators, etc. to access this data.

3) Split up of the MktData into MktDataSTK, MktDataFUT etc. The work-around for the "zero size bug" of Mike Smith does fit in this layer nicely.

4) a connection persistence and Resends the requests in case the link was broken with error (1101), 1102, 1300

5) A run method in the Connection class: i.e. Run(Run::Until(15,30,00)); which basically loops and calls checkMessages() and Connection::RunActions(). So my ATS or test programs have all a very compact 'main' routine: allocate the ATS and a call to the Run method.

I hope this view, although implemented in C++, is of any help. The implementation is not complete yet.

Jan Boonen

[Q] Python?

[A] For IB API + python awesomeness please check out:

Free udemy course: "Python programming with Interactive Brokers' Trader Workstation API"

https://www.udemy.com/course/python-api-for-trading/

Also:

For the Python language, Ewald de Wit has created an amazing framework called ib_insync that takes care of all the housekeeping and management functions and let’s you focus on your client code and trading. In case you are not aware of it, take a look at his GitHub repository and the extensive documentation. ib_insync allows you to operate the TWS API in its natural asynchronous request/response mode but also provides convenient blocking versions for more traditional sequential approaches. You can also use it as a rich source of examples on how you can achieve synchronization in Python.

Just take these two lines of code as an example. This is all you need in ib_insync to connect your client to the TWS API

ib = IB()

ib.connect('127.0.0.1', 7497, clientId=1)

Compare that with your code where you had to spend about 75% with boiler plate and connection related activities.

The ib_insync package: https://github.com/erdewit/ib_insync

and related discussion group:

https://groups.io/g/insync

Also: sprach groupse.io forum by word “jupyter”

https://groups.io/g/twsapi/search?q=jupyter

And then also:

https://groups.io/g/twsapi/message/37515

Use a GUI library like tkinter or pygame.  For testing you could also use jupyter,  there's a sample in here somewhere.

https://groups.io/g/twsapi/message/37329

https://groups.io/g/twsapi/message/37341

[Q] Delphi?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by josejp1

In this link are good examples of programming of Trader Workstation API with Delphi

http://delphimagic.blogspot.com/2010/07/interactive-brokers-con-delphi-tws.html

Ej. Connect with TWS, request market data, get ticks of price, place an order (mkt, lmt, stp, etc), request historical data, get information of an order, how to cancel orders

— also —

Delphi / BCB / Kylix TIABSocket component & API

http://www.hhssoftware.com/iabsocketapi/index.html

updated to the latest 9.76 API (January 2020)

This component set is built to work with the Interactive Brokers TradeWorkStation API. It allows Delphi and BCB programmers to build apps that send orders directly to the TWS. This TIABSocket component links directly to the socket to communicate with the TWS. There are no dll, ActiveX, .Net or DDE files required, and all socket messages & data processes instantly. This direct connection to the socket makes for a fast reliable interaction and response time. The component set also controls all interactions with the TWS, and includes sub components to manage order activity, portfolio and account details. The code looks and feels like any other Delphi component, and the syntax is similiar to existing standard component code. The complete source is included, along with a demo app and help file.

[Q] Java?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by Suminda

I have started a CEP (Complex Event Processing) based trading system which will be agile as possible. This is at a conceptualisation stage where we are running a few experiments on how the final architecture will be. Is it possible to help out in this as well as help build the team to take this forward.

Suminda

http://code.google.com/p/cep-trader/

java sidenote: by hubert.tse@gmail.com

My Eclipse was set on Java Compiler -> 1.5 Compliance Level.

Apparently 1.5 does NOT support @override and thus will not run. Once you set this to 1.6 and beyond, all the errors go away and it launches perfectly.

Many thanks for all your help!

[A] by Rob Terpilowsky (thread)

The 2 libraries I've used are

ta4j: https://github.com/mdeverdelhan/ta4j

and TA-Lib :http://www.ta-lib.org/

I prefer ta4j as it has a cleaner API than TA-Lib.

The 2 libraries I've used are

ta4j: https://github.com/mdeverdelhan/ta4j

and TA-Lib :http://www.ta-lib.org/

I prefer ta4j as it has a cleaner API than TA-Lib.

[Q] ruby?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A] by Dr. Hartmut Bischoff

Well, there is ib-ruby, too.

You can use pure ruby scripts, like

 require 'ib'

        ib = IB::Connection.new :port => 7496

        ib.subscribe(:Alert, :AccountValue) { |msg| puts msg.to_human }

        ib.send_message :RequestAccountData

        ib.wait_for :AccountDownloadEnd

        ib.subscribe(:OpenOrder) { |msg| puts "Placed: #{msg.order}!" }

        ib.subscribe(:ExecutionData) { |msg| puts "Filled: #{msg.execution}!" }

        contract = IB::Contract.new :symbol => 'WFC', :exchange => 'NYSE',

                                :currency => 'USD', :sec_type => :stock

        buy_order = IB::Order.new :total_quantity => 100, :limit_price => 21.00,

                                :action => :buy, :order_type => :limit

        ib.place_order buy_order, contract

        ib.wait_for :ExecutionData

https://github.com/ib-ruby/ib-ruby

Dr. Hartmut Bischoff

Stuttgart

Update 06-Jun-2018: by Dr. Hartmut Bischoff:

View/Reply Online (#39948) 

(quote)

Imagine you got a trading robot at a remote location, accessible only via an reverse ssh tunnel, or a TWS is running on a cheap cloud-server.

Simple-Monitor ( https://github.com/ib-ruby/simple-monitor ) is designed to display the status of a Friends&Family-Account via web-interface.

It comes with just 435 lines of code using the camping micro-framework and ib-ruby.

Its intended to run even on resource-low  hardware and serves well running elinks in a tmux-session, avoiding a running TWS-GUI in favor of the TWS-Gateway

It too demonstrates how well ib-ruby ( https://github.com/ib-ruby/ib-ruby ) integrates in existing frameworks.

(end of quote)

[A2] by Peter Gum

IBCsharp, tradecommander, openquant,

tradelink and others. RighEdge <http://www.rightedgesystems.com/> is as

comprehensive as any, including supporting VB.NET

google "jTWScharts"

[Q] C#?

Try to check “Files” section of TWSAPI Yahoo Group. Also:

[A1]

see: Interactive Brokers C# Api (by Karl Schulze)

ib-csharp

https://github.com/krs43/ib-csharp

and/or:

http://www.dinosaurtech.com/utilities/

[A2] by Rich Holowczak (a.k.a. “Prof. H.”)

I have written a few more to work with IB's C# API (console only applications). same location:

http://holowczak.com/category/ib/

Cheers

Rich H.

if you aware of any other published working code examples, please let me know so I can add the link into this FAQ

[Q] R

[A] by ce (from this thread)

pdf files in

http://cran.r-project.org/web/packages/IBrokers/index.html

are good.

 

Also there is a very good example system in

http://censix.com/download/

[A2] by censixtws (from this thread)

Hi I have developed this collection of R scripts that I call qsiblive ( downloadable from my website http://censix.com ) and am currently testing it with my paper account. It trades one symbol on an EOD basis. I think its strength lies in the immense flexibility that it provides with regard to 'plugging-in' R analytics packages/methods (some classic indicators, some ARMA, GARCH modeling, you name it). Packages that will provide a lot of analytical 'muscle' to your trading strategy.

I am making this available to get some feedback from the community of seasoned traders/developers here.

Cheers

Soren

see:

censix – pure R intraday trading framework – screencast01

https://www.youtube.com/watch?v=Zzd7tx452Ao&index=2&list=LLYax4ZgrNYHP5fLotKBc9Lg

censix on GitHub: https://github.com/censix/INTRADAY-PartAB

[R-related resources]

Contemporary Portfolio Optimization Modeling with R

https://www.youtube.com/watch?v=hKDalfhDawA

[Q] Pattern recognition library?

Added on 16-May-2017

is there an open source library for chart patterns? or at least algorithm so i can use to program it myself?

looking for flag, triangles and double top/bottom mostly.

[A] by Johnny Red (View/Reply Online (#37504))

I use TA-Lib – it's open source and very good.  It's written in C++ and is available for Java/C#/etc.  I even think there is a version for Python.

Good stuff.

http://ta-lib.org/hdr_dw.html

[A2] by Peter Cawthron

There’s a Python wrapper for TA-Lib at https://github.com/mrjbq7/ta-lib

Regards,

Peter

[Q] Reverse engineering java?

[A]

I know you mentioned you've resolved your issue, but just FYI, if you want

to peek at the contents of "other.jar":

http://java.decompiler.free.fr/

Grab the GUI version and click "File->Open", pick "other.jar" and browse the

source. The functionality is essentially the same as the .Net Reflector.

–s

ps. if reverse engineering is illegal in your country, you didn't hear this

from me. (I'm Canadian, things are pretty lax up here.)

——- also ——–

Alternatively, make a copy of it and rename to "other.zip" and open as

usual with Windows or your Mac.


Implementation notes

[Q] How do I retrieve the bid/ask spread for given contract?

[A] View/Reply Online (#46694)

Added on 13-Mar-2021

I was working on it recently, so let me try to put a 1st draft of the answer:

[Q] How do I retrieve the bid/ask for an option leg?

[A] I found 3 ways so far. Only one works for me, but I'll mention all 3 here.

1-of-3: Using client.reqTickByTickData(19003, ContractSamples.USStockAtSmart(), "BidAsk", 0, true);

Read this article: TWS API v9.72+: Tick-by-Tick Data

In particular read about IBApi::EWrapper::tickByTickBidAsk, which would give you bid/ask.

The problem here is that you only allowed only 3 simultaneous subscriptions, which might work for someone who's training 1-2 instruments, but I wanted to find more generic way to get bid/ask for more contracts at the same time.

2-of-3: Using EClient.reqMarketDepth (reqMktDepth in Java, C++, and Python). See these 2 docs:

TWS API v9.72+: Market Depth (Level II)

TWS API v9.72+: EClient Class Reference

In their docs IB says:

Beginning in API v974 and TWS v974 it is possible to Smart-route a IBApi.EClient.reqMarketDepth request to receive aggregated data from all available exchanges, similar to the TWS BookTrader display.

(end of quote)

And I come to the conclusion that this "aggregated data" is the only right way to get proper bid/ask unless you're always trading some contract that is only traded on 1 particular exchange and thus you don't need to aggregate data from many exchanges. If you use "reqMarketDepth" with contract.Exchange set to SMART and set "isSmartDepth" argument to true, then you'll get an error, that you require more subscriptions. In my data subscriptions I didn't have any Level-II except a free one for "IEX exchange". If I try to set contract.Exchange = "IEX" and set "isSmartDepth" argument to false, then I start getting data, but noticed, for example for "IBM" contract, that the bid/ask during any quiet low volatility day (shown in TWS around $0.02), is reported by IB API often times as a much bigger number (for example $2.45 vs $0.02 expected – see screenshot below). IB support confirmed that what you see in TWS bid/ask (with list of different exchanges) is NOT accessible via API without subscription, but is accessible for your "human naked eye" for free in TWS window (it's time to engage OCR!-)).

Here's an example when IB API returned $2.45, while TWS shows aggregated data with bid/ask = $0.02 around Feb 8, 2021, 2:47 PM

You can see IEX Bid is 121.0 and IEX Ask is 123.45, so the spread by "IEX only" version is 2.45! Nasty and useless!

If you want to use SMART for the "reqTickByTickData" you'll have to have Level-II subscriptions for ALL exchanges this contract is traded on.

To check how much all of them will be login to interactivebrokers.com -> AccountManagement -> Settings -> User Settings -> Market Data Subscriptions ->

You'll see "North America: IEX Depth of Book – Trader Workstation    – Fee Waived"

Along with notes:

IEX exchange depth of book service. Requires prior subscriptions to NYSE (Network A/CTA), AMEX (Network B/CTA), and NASDAQ (Network C/UTP) or US Equity and Options Add-On Streaming Bundle.

(end of quote)

Then click: Current GFIS Subscriptions -> configure gear icon -> North America (as example) -> Level II (Deep Book)

And you'll see the list with prices.

3-of-3: poor-man bid-ask using real-time bars and "ticks"

If you subscribe to two real-time 5 seconds bars streams by either:

  a) IBApi.EClient.reqHistoricalData request with "keepUpToDate" set to true (see also TWS API v9.72+: Historical Bar Data)

    or

  b) IBApi.EClient.reqRealTimeBars request (see also TWS API v9.72+: Historical Bar Data)

And specify "what to show" to "BID" for one request and "ASK" for another request, then IB API will call back with 5-second bar updates every few seconds (on average every 3.5 seconds) and each stream "bar.close" price will represent most recent bid and ask. The only problem is that those 2 streams represent 2 events streams (separated in time). I filter out negative calculated bid_ask spreads as well as calculate bid_ask spread only if bid update and ask update are within 500ms distance. As a result on the same "IBM" ticker I got on average (during the whole trading day) calculated bid_ask values coming on average every 3.5 seconds, which is good enough for me 🙂

Note #1: this method is only counted as "one line" out of your 100 available lines, so theoretically you can get 100 bid_ask spreads (along with sampled "real time tick" for price updates) without buying any data-boosters or Level-II subscriptions.

Note #2: yes, there is "what to show" type "BID_ASK" (see Historical Data Types), but it will not give you an instant snapshot of the current most latest bid and ask values. Instead you'll get:

bar.open = "Time average bid", bar.high = "Max Ask" (useless), bar.low = "Min Bid" (useless) and bar.close = "Time average ask". I'm not sure if the "time average" values for 5-sec bars will be better than the above method, I'll try to check them side-by-side, but my guess is – both approaches will yield similar quality output.

I'm not sure how well this method would cope with fast moving markets. I guess it will give quite a random estimate for what really is. But one can always use some safety factor (for example use calculated bid_ask spread value and multiply it by 2.0 just to be sure). Also it would be interesting to see how the spread value correlates with the current volatility. During volatile times the price might jump, the bid/ask might jump, but the high volatility reading will be showing pretty much flat high value, which you can use to estimate the bid/ask max value (and/or to adjust the "safety factor" and make it a bit higher when market goes crazy:).

There's some free 15-minutes delayed bid_ask stream if I recall correctly (as 4th option), but I haven't looked into that option since the 15 minutes delay is a bit too far.

Cheers,

Dmitry Shevkoplyas

ps: I'll show here the function I used on every incoming bar to prototype the bid_ask calculations, think of this piece as "meta language":

   public void on_historical_bar_try_to_calculate_bid_ask_spread(

           Bar bar,

           RequestHistoricalData request) {

       // Check we have both requests in place (not nulls)

       if (request_historical_data_bid == null) {

           logger.debug("request_historical_data_bid == null, nothing to do yet…");

           return;

       }

       if (request_historical_data_ask == null) {

           logger.debug("request_historical_data_ask == null, nothing to do yet…");

           return;

       }

     

       // Both bid / ask "real time" bars requests are present! We can calculate bid/ask spread estimate.

       // Let's extract most recent bars from 2 streams:

       Bar bid_bar = request_historical_data_bid.most_recent_bar_received;

       Bar ask_bar = request_historical_data_ask.most_recent_bar_received;

       if (bid_bar == null) {

           logger.debug("bid_bar == null, nothing to do yet…");

           return;

       }

       if (ask_bar == null) {

           logger.debug("ask_bar == null, nothing to do yet…");

           return;

       }

       // Check how far in time they are from each other

       long bars_received_time_delta_ms = request_historical_data_bid.most_recent_bar_received_time.get_delta_ms(request_historical_data_ask.most_recent_bar_received_time);

       // Filter out updates with more than 500ms distance

       if (Math.abs(bars_received_time_delta_ms) > 500) {

           return;

       }

       // Both bid/ask bars are present and close enough, let's calculate the bid/ask spread

       double bid_ask_spread = ask_bar.close() – bid_bar.close();

       // Filter out some obvious BS

       if (bid_ask_spread <= 0) {

           return;

       }

       logger.info("Calculated synthetic poor man's bid_ask_spread: "

               + Aid.math_round(bid_ask_spread, 2)

               + " bars_received_time_delta_ms: " + bars_received_time_delta_ms);

   }

[Q] How to close an OCA group?

[A] View/Reply Online

Added on 22-Feb-2021

Nick: You can also modify the stop order to be a market order. This may not work with all order types so you have to try it with your scenario.

[nice and elegant solution!]

ds-avatar: TWS will automatically assign an OCA (one-cancels-all) group id to the child stop loss order. That's how it implements the execution of only a single child order if more than one are submitted. You can use it for your manual market exit order so its execution will cancel the child order.

[Q] Order field to identify a strategy? (clientID vs orderRef vs permID)

I have multiple strategies running on the same account/sub-account.

I want to distinguish which orders were placed by who.

Is there a field in the Order class that I can assign a custom value, and that value to be persistent so I later can identify who placed the order?

Someone suggested to use orderRef field.

Similar question: by Corneliu Maftuleac corneliu.maftuleac@gmail.com View/Reply Online (#46716)

I am using the orderRef field of an order to mark an order as it belongs to a specific strategy.

However when using flex queries to retrieve the list of orders the OrderReference field is always empty.

As specified on the docs: The order reference number as defined by the user on the order ticket.

Any ideas how to retrieve order reference?

[A] View/Reply Online

Added on 22-Feb-2021

There is OrderRef but it might not persist across sessions, you would have to make sure it meets your needs.

By "Session" Nick meant:

Except for forex, markets are open for only part of a day as opposed to continuous 24/7 trading.

The general idea is markets open at a specific time and then close at a specific time and that is one trading session. Trading on the next day is a different session.

Different financial instruments have different rules for when a session starts and ends. A single session on the major futures exchanges spans calendar days – opening in the evening one day, going past midnight until the afternoon or the next day. Can make you crazy since a trade can occur on Monday evening but counts as the Tuesday trading day.

Back to the OrderRef, people have observed that the field is returned for executions in the same session as the order was placed but is empty if the order executed in a subsequent session.

Mark Colling also suggested OrderRef:

I use trade references stored in the order reference field, it usually works without problem. Track the order id, order reference and your internal tracking identifier, and you should be able to piece it all together.

What I have experienced is position reports being lagged by many minutes, and execution reports arrive in an arbitrary order, but the order references do seem to be stable.

Answer by ds-avatar:

OrderRef will be in reports only for the most recent session. Workaround: configure the system to send daily reports by mail, then collect and collate them.

[best answer by praditik]

I have used permID and stored it locally to identify strategy mapping. It works seamlessly across sessions and days. So as everyone else already said, I think, using permID with local mapping in your program should be able to do the trick.

Resolution:

Yes, the OrderRef is unfortunately eventually gone, but the report should also carry order.permID field, which is account-unique identifier, and it should be there forever.

So you can keep track locally of all your ordersRef values and corresponding permID assigned to your order. Thus many months after, when you pull the report from IB you'll be able to restore original orderRef value by lookup into your local table (orderRef <–> permId). I agree – the IB's report should be self-sufficient and I don't understand the reasoning of deleting some of the existing fields (if it is there – it should be there forever), but it is what it is. Please let me know if it works for you? Do you see permID in the report?

Corneliu Maftuleac: – Yes, we see permID in the report.

[Q] Detection of loss of connectivity

[A] View/Reply Online

Added on 04-Feb-2021

Eugene,

Yes, there was very similar topic discussed around September 2015 in this thread (subject: asynchronous EClient::eConnect() needed?) and the conclusion was that if you just connected to TWS API and don't send anything then you might get hard time detecting it quickly (and in some cases you'd even get stale connection on the TWS side and won't be able to re-connect to TWS using same client ID without restarting it (or after something like 30-40minutes)).

The solution to this would be to subscribe to something "noisy enough" so TWS site would try to use established tcp connection to your client app as well as in parallel keep asking from your client to TWS API something simple, like reqCurrentTime() every second (TWS answers it on its own, without bothering/asking remote IB's servers, so it is safe to be annoying:)   Those 2 "streams" will be the "heartbeats" you're looking for!

hope it helps,

Dmitry

Hi Kurt,

Seems in my 1:30am reply I slipped to the "client <==> TWS" detection instead of "TWS <==> IB".

As Eugene mentioned api client only detects connection lost when it tries to send order (or perform other api call to TWS), which is exactly the problem I observed about 1yr ago in mentioned above thread when my client was running on other than TWS box. So to mitigate it (and make api client aware about its connection lost to TWS quicker) client should try to talk to TWS. Also TWS clears its internal connection-describing tables much quicker if it fails to deliver something to client hence suggestion to tickle both directions by asking time (client -> TWS) and subscribe to some ticker (TWS -> client).

As for "TWS <==> IB connection lost" while "client <==> TWS" connection is fine (say for simplicity api client and TWS are on the same host) then TWS indeed have some mechanism (like heartbeats) – even when you aren't subscribed to anything in TWS you can see every 10sec remote IB host (in my case cdc1.ibllc.com, which also can be found in ~/Jts/jts.ini) pushes packets with content like this (truncated packets payload): "FIX.4.1.9=00014.35=1.112=farm.10=217" and

"FIX.4.1.9=00027.35=0.112=FixTestRequest195.10"

and then TWS reports back to IB something FIX-protocol-related, like:

"FIXCOMP.9=70.x"

And when "TWS <==> IB connection" is lost then api client is notified by TWS immediately. So luckily for us there's no need to invent a new magic wand to cause TWS to "share" with the client app information previously unshared about its awareness of its connection with IB – this logic is already there. My TWS reports connection lost within 30sec and recovers within 5-10sec upon network restored.

Sorry for not being clear with my midnight statement 🙂

thanks,

Dmitry

[Q] Get list of all symbols by exchange?

[A] View/Reply Online (#46323)

Added on 29-Jan-2021

NASDAQ  has API, which gives you list of all symbols and even their last price as JSON:

https://api.nasdaq.com/api/screener/stocks?tableonly=true&limit=25&offset=0&download=true

[Q] Getting stocks / options standard deviation?

Is there a way in TWS to get a simple option analysis graph?

Do you know how to calculate the stock price standard deviation?

The goal is to get via API all necessary data to calculate this SD for any stock/options.

[A] View/Reply Online (#44329)

Added on 03-Oct-2020

Take a look at HISTORICAL_VOLATILITY on reqHistoricalData.

https://interactivebrokers.github.io/tws-api/historical_bars.html#hd_what_to_show

— also

In TWS, you can right clight on an option and ask for its performance profile, it's almost the same window.

If you use reqMktData on a stock contract with IDs 104 and 106 you will get annualized HV and IV for a specific stock

You can fetch annualized IV for a specific option contract with the ID 106, but not the HV

You could also compute the HV by yourself if you have more than 30 ticks, as I believe IB uses Garman Klass formula with a window of 30. But you can use any other volatility formula I suppose.

Of course to get an accurate range for a given date, you would need to adjust the volatility to the period of time. If you want the volatility over 15 open days: VOL / ((15/252) ** 0.5)

Hope this helps

[Q] How to detect TWS disconnected from API?

On Wed, May 20, 2020 at 02:45 PM, Spread Haendler wrote:

I am using TWS 978.1s and latest TWS API 9.79.01 (hopefully latest API version).

If the TWS disconnects from API (unfortunately I have frequent sudden shutdowns of TWS978), I receive no error message via API.

[A] by rwk, View/Reply Online (#44515)

Added on 04-Aug-2020

It seems risky to depend on a disconnect message to signal that something has gone wrong.  We could have a good connection and still not get quotes or be able to place orders.  What I do is monitor a contract such as an index ETF that I know should be active.  After some interval, such as 20 seconds without data, I raise an audible alarm and investigate.

[Q] What value means "no value"?

[A] by Kurt (src)

IB shows no consistency in the use of values meaning "no value".  Sometimes it is INT_MAX or DOUBLE_MAX, sometimes 0, sometimes –1, sometimes –0.99, etc.

The historical bar volume field is only populated for TRADES, never for BID, ASK, MIDPOINT, BID_ASK, OPTION_IMPLIED_VOLATILITY, or HISTORICAL_VOLATILITY.  Instruments that do not provide TRADES (such as FX) therefore do not have volume available.

There are probably many other things omitted for various instruments in ways that follow no particular pattern, and are not documented.  You just get what you get.  Don't worry about things looking strange.  That is the least of the problems you will have at IB.

I've definitely noticed that in historical data the unused fields in a bar are sometimes 0 and sometimes –1.  There is definitely no documentation about that.

–Kurt

[Q] Best practice to handle ID’s / nextValidId?

[A] by edbar@exmsft. from this thread

I store the ID on my hard drive.

Then when I start up TWS, I check the nextID that they send to see which one

is bigger.

If my stored ID is greater than the TWS number then I go with mine.

Then, every time I increment the ID number, I update my hard drive backup.

You can always go to the TWS CONFIG API screen and reset the api id numbers.

–Ed

[A2] by btw12342001 (same thread)

I don't think you should use the orderId for tickerId. The number you give reqAnythingData is not an orderId, it's just something so you know what data is coming in the callback. It's possible you subscribe to more than one symbol of data. It still doesn't explain why you're mixed up with orderId's

I guess I don't need to answer the 2002 question 🙂

[A3] by Kurt (same thread)

A lot of people have reported here that just using nextValidId() is not

sufficient. I've been doing it without problems so I may not recall the

details, but I believe the common practice it to keep track (on disk) of

your greatest order id, and use the greater of that and the value returned

by nextValidId when you start up, and increment from there. You can check

the archives. This question was last asked less than 2 months ago I

believe.

Regarding reqMktData etc each one has its own id space but for your own

sanity you probably want to keep them unique. I keep all request id's

unique *except* for placeOrder for which I used a separate scheme. Using

unique request ids facilitates routing incoming request errors.

-Kurt

[A4] by Richard L King rlking@aultan.com View/Reply Online (#44680) 

Added on 10-Oct-2020

There is no requirement imposed by the API to use different ranges of ids for the various requests that use them (about the only rule is that order ids must never decrease, except that you can reset the order ids for a particular clientid via TWS). Moreover, there’s nothing to prevent you using the same id for a market data request, a historical data request, an order etc all at the same time.

 

As you’ve observed, this causes potential problems with the error callback.

 

One way to address this is to define your own ranges. For example, 0-1023 for market data requests, 1024-2047 for contract details requests, and so on. You of course have to think about what happens when you get to the end of a range, and there are many ways of dealing with this, and such considerations will guide you in deciding how big the ranges should be. For example, contract details requests complete fairly quickly, so you could almost certainly get away with a range of say 1000 ids for them, and just wrap back round to the beginning when you get to the end – the chances of the first request not having completed by the time you’ve made the next 999 requests is remote: however I personally wouldn’t take the risk, and I’d use some kind of data structure to keep track of ids which had been used and those which were free.

 

If you do this, then it’s trivial to decide, for any error message that includes a non-negative id, which type of request it relates to, and even which precise request if, for example, you use the ids to index into a table per request type.

 

I use such a scheme. In particular I use 0x10000000 to 0x7FFFFFFF for order ids. This large range means I’m not going to run out of order ids in my lifetime (there are 1,879,048,191 of them!), so I’ll never have to use the order id reset mechanism. I also ensure that within this range, ids for single orders and entry orders in brackets end in 0 or 5, stop-loss orders in brackets end in 1 or 6, and take-profit orders in brackets end in 2 or 7: this is simply to make it easier to identify the role of an order when looking at a logfile.

 

A scheme of this sort should be entirely compatible with your use of CompletableFutures.

 

Richard

[Q] After connection loss I need a list of all open orders and positions in order to get synchronized?

[A] by src

Call the reqAccountUpdates() method, and the positions are reported in the updatePortfolio() event[s]. The events fire once for each position. Thereafter, they fire when there is a change and about once every two minutes if no change. It works well.

[rwk]

You might consider also requesting executions. Sometimes open orders or

execution reports (or both) may fail to work right, due to IB misfeatures or

whatever. The more information the better, so that means requesting

positions is also a good idea. Positions are part of account updates.

-Kurt

> What do I send in the acctCode parameter of reqAccountUpdates()?

You can set it to the null string if you do not have an FA account scenario.

-Kurt

or you can set it to: your account code UXXXXXX or FXXXXXX

> What do I receive in the position parameter of updatePortfolio()? Is it the

number of contracts in the Portfolio

YES

[Q] Tracking time (internally)

[A] by Dmitry

This has nothing to do with any incoming timestamps from outside – it is just for local timestamping of all internal events. As the good enough resolution I choose millisecond resolution. Local time is kept my NTP with updates polled from ntp servers as often as we can (had to build own Stratum-1 ntp server based on gps pps signal (see Time drift for more details))

long get_epoch_ms()

{

    // http://brian.pontarelli.com/2009/01/05/getting-the-current-system-time-in-milliseconds-with-c/

    timeval t;

    gettimeofday(&t, NULL);

    long epoch_ms = (t.tv_sec * 1000) + (t.tv_usec / 1000);

    //std::cout << ";" << t.tv_sec << ";" << t.tv_usec << " or simply epoch_ms: " << epoch_ms << std::endl;

    return epoch_ms;

}

[Q] Building live Trades

see also relevant: [Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] What is the sequence of tickSize() tickPrice events?

[A] by Dmitry

If you want to extract events for closed deals (not some BID/ASK moves noise, but really closed deals with real price/volume), then you gotta follow these considerations since it might be not quite straight forward due to IB would only send relative “differences” instead of absolute self-contained discrete messages about trades. To warm up, please read: [Q] Sometimes lastSize=0. wtf?

Now the implementation algo to get trades Dmitry uses:

– have container for  trades within MyContract class

– all tickPrice(), tickSize() events are routed by ewrapper to request object (RequestMktData), which is associated with MyContract class instance, which in turn has tickPrice() tickSize() methods to process incoming ticks

– now inside MyContract class:

void MyContract::tickPrice(Account *account, long epoch_ms, TickerId tickerId, TickType tickType, double price, int canAutoExecute) {

    LOG(INFO) << "MyContract::tickPrice";

    // forward raw tick to subscribed algos (if any)

    for (std::vector<AlgoBase*>::iterator ialgo = subscribed_algos.begin(); ialgo != subscribed_algos.end(); ialgo++) {

        (*ialgo)->tickPrice(account, this, epoch_ms, tickerId, tickType, price);

    }

    // analyze if GotTrade()

    if (tickType == LAST) {

        // we got new trade, but we have to wait for TickSize() call to

        // complete the Trade obj (and then will call GotTrade(trade) to pass

        // complete Trade object to all subscribers)

        this->tradesPerAccountMap[account_as_key].push_back(new Trade(account, epoch_ms, tickerId, price));

    }

. . .

}

void MyContract::tickSize(Account *account, long epoch_ms, TickerId tickerId, TickType tickType, int size) {

    LOG(INFO) << "MyContract::tickPrice";

   

    // forward raw tick to all subscribers

    for (std::vector<AlgoBase*>::iterator ialgo = subscribed_algos.begin(); ialgo != subscribed_algos.end(); ialgo++) {

        (*ialgo)->tickSize(account, this, epoch_ms, tickerId, tickType, size); // forward tickSize() event to subscribed algos (if any)

    }

        // analyze if GotTrade()

    if (tickType == LAST_SIZE) {

. . .

        // CASE 1 of 3: check if last_trade is in "waiting for size" state (last_trade->size < 0 aka 'uninitialized'))

        if (last_trade->size < 0) {

            // last_trade's size is not yet initialized, means we waited for this tickSize() event

            // to complete the trade obj (and forward it to all subscribers)

            last_trade->size = size;

            last_trade->completion_time_ms = epoch_ms – last_trade->epoch_ms; // we track how long it took us to complete this trade object

            last_trade->is_complete = true;

            // forward complete trade to all subscribers

            gotTrade(last_trade);

            return;

        }

        // CASE 2 of 3: Is it duplicate of last_size tick (same size as last complete trade and within 9ms) => just ignore it

        if (last_trade->expect_dup_ticksize && last_trade->size == size && (epoch_ms – last_trade->epoch_ms) <= 9) {

            last_trade->expect_dup_ticksize = false; // mark last_trade as "got expected duplicate" so next tickSize if

            // happen to arrive within <7-8ms with the same size field would not be treated as duplicate anymore

            // (the 'expect_dup_ticksize' flag allows us to ignore tickSize only once

            // and only for expected ticks (ones that go after "Last" 2 times in a row))

            return;

        }

        // case 3 of 3: It is normal LAST_SIZE tick telling us about new trade with the same price as before! We got new trade!

        Trade * new_trade = new Trade(account, epoch_ms, tickerId, last_trade->price, size, true, 0, false);

        ptrades->push_back(new_trade);

        // forward complete trade to all subscribers

        gotTrade(new_trade);

        return;

    }

}

[Q] Building live bars

[A]

https://groups.io/g/twsapi/topic/4046604

https://groups.io/g/twsapi/topic/4046154

[Q] Why do we get two identical tickSize() callbacks?

[A] explained by Richard L King

Date: Sat, 5 Feb 2011 00:47:45 -0000

Subject: RE: [TWS API] Re: tickSize(): LAST_SIZE and VOLUME order

Perhaps I can throw a little light on this subject to explain what is

actually happening here and why.

In the early days of the API the TICK_PRICE protocol message (ie the message

itself, not the callback function/event) had a version number of 1 and had a

price field, but no size field.

So at that time, all prices came in TICK_PRICE messages and all sizes came

in TICK_SIZE messages. Bear in mind the sampling mechanism that IB uses,

which only sends values that have changed during the previous sampling

interval. If a price had changed, it would send a TICK_PRICE message before

the corresponding TICK_SIZE message. But this approach had a problem,

because a TICK_SIZE message was only sent if the size had changed. Thus

there was no way of the client knowing, when a TICK_PRICE message arrived,

whether the corresponding size was the same as the previous size (in which

case there would be no TICK_SIZE message) or whether it too had changed and

a TICK_SIZE message would follow.

To solve this problem, IB introduced version 2 of the TICK_PRICE message in

API v760 Beta (my copy of this is dated Feb15 2004). This included a size

field, and now whenever a TICK_PRICE field was sent (which meant a change of

price during the sampling interval had changed), the size could be included

whether or not the size itself had changed. The TICK_PRICE message would now

generate both a tickPrice() callback followed by a tickSize callback. This

meant that clients had it nice and easy: when you got a tickPrice() callback

you just had to remember the price, and any following tickSize()s would

implicitly relate to that price until the next tickPrice().

However there was now a different problem. Any client that was using an

older version of the API with client version less than 15 would not

understand the version 2 message and so TWS would continue to send version 1

TICK_PRICE messages to those clients. But this meant that the tickSize()

callback following the tickPrice() callback would not be generated, so the

client would not have the full information. To fix this, after sending the

TICK_PRICE message, IB decided to also send a TICK_SIZE message with the

same size as the TICK_PRICE message. Thus clients with version less than 15

would now receive the full information.

The stupid thing was that IB should have only sent the extra TICK_SIZE where

the client version was less than 15, but they actually sent it to all

clients, and of course they still do. So now we have this silly situation

where after every TICK_PRICE message a redundant TICK_SIZE message is sent,

resulting in a tickPrice() callback followed immediately by two identical

tickSize() callbacks.

So why don't IB fix this? Well, I don't really know. It may be they think

it's of no significance, because for example anyone watching the data

onscreen would never notice. It's only because we like to tally things up

and compare with the cumulative volume figures that it irks us (and of

course because it's just plain stupid!).

And once you understand what's going on it's easy enough to code round it,

as Mike has already pointed out below.

If anyone wants to complain to IB about this and persuade them to fix it, by

all means have a go, but I very much doubt whether they'd show any interest

at all. They haven't for the past 7 years (and after all they themselves

must have had to code round it in TWS).

Actually the real annoyance here is not so much the issue itself, as the

fact that so many people spend so much time discovering it, worrying about

it, and inventing workarounds. But then that's the case for so many aspects

of the API…

Richard

[A] explained even more by Richard L King (found here)

Added on 09-Jun-2015

Let’s summarise what’s going on here:

1. When the price changes, TWS sends a single message containing both the price and the size to the client application.

2. IB’s API code then makes separate tickPrice and tickSize callbacks (note that the latter is immediately after the former – you can easily see this in the API code.

3. IB’s servers then send the size again, and the API makes another identical tickSize callback (the reasons for this are explained in my contribution to the FAQ linked to by rholowczak’s post on 2 June – this is the part highlighted in blue: I suggest you read that carefully).

4. Each time there is a trade with the *same* price but a *different* size, another tickSize callback is made.

5. When there is a trade with the *same* price and the *same* size, no tickSize callback is made.

From this you can immediately deduce that there will be *at least* 2 tickSize callbacks per tickPrice callback, and usually more because the price doesn’t change with every trade.

By the way, there is no need to fiddle around with ‘blanking time’ that you mentioned in your previous post. If you get a tickPrice callback, just record the price. The very next callback is *always* the tickSize (check the API code to convince yourself that this is true), so record the size and call your application code to process the combined price/size. Subsequently whenever you get a tickSize callback, if the size is the same as the last size you recorded, just ignore it (because it will be the duplicate tickSize described in 3 above), otherwise call your application code to process the combined price/size where the price is that recorded in the latest tickPrice callback.

It may be worth pointing out that in spite of what I said in 5. above, there *are* occasions where a tickSize occurs where the size is the same as the previous one and the price has not changed – however they are rare, and given the approximate nature of the data feed it’s not worth trying to complicate things to try and detect them. Having said, that my own API code does complicate things somewhat, but I’m not sure what I gain by doing so. So I won’t bother to describe these extra wrinkles unless someone is really interested.

see also somewhat related:  [Q] Building live Trades

[Q] Could someone suggest a good algorithm for receiving market data for more than 100 symbols with a single account?

[A] by Jason Sapp

I use a "round robin" type of algorithm that utilizes one of my paper trading

accounts to obtain cumulative volume on about 1000 symbols. I am not too

concerned if my cumulative volume number is off by a couple minutes, so this

works for me. The way I do it is as follows:

-Call reqMktData (which contains cumulative volume) for the first 90 symbols in

my list (making sure not to call reqMktData more than 30 times per second—at

least I think 30 was the magic number, but this limitation may longer be in

place).

-Wait 10 seconds (after calling reqMktData on my the last of the 90 symbols).

-Call cancelMktData on all of those symbols.

-Obtain the next 90 symbols from my list.

-Start over.

The reason I used 90 symbols was because I sometimes had a market row or two

being used by my paper trader TWS.

Jason

————————- also alternative ———————————

from here

I use Yahoo Financial's CSV feed b/c I can request multiple symbols within a

single request to get quick price data snapshots for each symbol. For

example, to request last price, daily high, daily low, close, and volume for

2000 symbols, it takes like 10 seconds to get that information from Yahoo.

I use this specifically in the morning when scanning through stocks. This

is more efficient, quicker, and reliable than trying to round-robin my way

through TWS requests and dealing with the request limits they have on their

accounts.

…   by Brant Hahn

[Q] data links drop; feeds fade. Serious software needs to handle that. How?

[A] implementation idea by Russ Herrold

As a practical matter, by subscribing to some indicator that

is continuously updated as well as a more leisurely symbol,

one always has some value appear during that second when the

link is live — watch ES or such 😉

[Q] Request/response/error architecture thoughts?

[A] by Kurt (from this thread: C++ architecture question)

There are hundreds of ways of doing it.  I don't believe in "ideal".  I'm posting some code here because it might be useful for some others as well as you.  (From this code you could probably reproduce a big chunk of the request/response/error architecture I use.)

I implemented request tracking classes with an instance per request.  The default behavior is for the constructor to send the request but it is also possible to create the request and defer sending it.  The class provides a bunch of useful functions including tracking latency, logging, and matching responses and errors to requests.

For this purpose I have an ActiveRequest class with subclass ActiveRequestWithContract for contract-based classes, adding additional information to the generic logging capabilities provided by ActiveRequest.  Then I further have a separate subclass for each IB request type which is either a subclass of ActiveRequest or of ActiveRequestWithContract.  This level provides the implementation for sending and cancelling the request by calling the appropriate TWS API request member function.

I use further subclassing for each distinct purpose or for distinct interfacing requirements, such as dispatching into Objective-C handlers.

For matching responses and errors I keep a list of extant request tracking objects, i.e. those objects that have not been deleted, which is an independent consideration from whether they have been sent, have received a response or a full set of responses, etc.  The ultimate client of the request can determine whether the request object should be deleted when the response is complete, if applicable.

As a result there is a list of existing requests which can be searched by request id.  This permits incoming responses and incoming errors that reference request id's to be routed to the appropriate request tracking object.  Thus request clients can be notified when a request is aborted due to an error.

This is not optimal, so perhaps deficient and certainly not ideal.  If performance were an issue the list of requests could instead have been a B-tree, or even a hash table.  So you could use the associative arrays provided by the C++ STL.

In any case the bulk of request tracking implementation is quite removed from the tickPrice callback.  However for each request tracking subclass that is specific to a particular IB request type, I have a method (member function) with name and parameters identical to the corresponding EWrapper member.  I just copy/paste the function prototype.  Thus a part of the class hierarchy looks like this:

    MarketDataRequest : ActiveRequestWithContract : ActiveRequest

and while my EWrapper subclass of course has this member

   virtual void tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute);

my MarketDataRequest class also has

    virtual void tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute) = 0;

for which the implementation is left to the next-level subclass.  This approach gives me a generic interface to the TWS API with none of the specifics of my app present in it.

The generic tickPrice implementation then does a linear search of a doubly-linked list to find the request object to route to, and looks like this.  The superclass of MyWrapper provides a default implementation that just logs the request.  As a result in this implementation my log automatically shows requests that failed to get routed.

void MyWrapper::tickPrice( TickerId tickerId, TickType field, double price, int canAutoExecute)

{

    for (ActiveRequest *req = marketDataReqTracker.Next(); req != &marketDataReqTracker; req = req->Next())

    {

        if (req ->ReqId() == tickerId)

        {

            if (MarketDataRequest *specificReq = dynamic_cast<MarketDataRequest *> (req))

            {

                specificReq ->tickPrice (tickerId, field, price, canAutoExecute);

                return;

            }

        }

    }

   

    inherited::tickPrice (tickerId, field, price, canAutoExecute);  // default logging used when response not routed above

}

It is not perfectly efficient, but it is "perfectly adequate" for my purposes and easy enough to upgrade if needed even though I am not using container abstractions (STL or otherwise).

The use of dynamic_cast is strictly unnecessary, because marketDataReqTracker should reliably contain only MarketDataRequest objects.  So this code could have benefited from some template use.  But even without that a static_cast would have been safe and only because the performance hit is no issue I just did the conservative thing.  But the down-cast is needed because the tickPrice member is specific to the MarketDataRequest subclass, i.e. the entire EWrapper interface is definitely not contained in each request object, per the description above.

Meanwhile error handling somewhat parallels this, with my EWrapper subclass providing this implementation

void MyWrapper::error(const int id, const int errorCode, const CString errorString)

{

    (void) error_Route (id, errorCode, errorString);

}

and error_Route special-cases a bunch of error codes for more sensible and direct routing and otherwise calls ReqTracker_error_Route:

ActiveRequest *MyWrapper::error_Route(const int id, const int errorCode, const CString& errorString)

{

    if (errorCode == 505)

        // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

        printf ("generic error %d %s [id=%d]\n", errorCode, errorString.CStr(), id);

    else if (id == kIBNullReqId)

        printf ("generic error %d %s\n", errorCode, errorString.CStr());

    else

        printf ("request %d error %d %s\n", id, errorCode, errorString.CStr());

   

    bool handled = false;

    ActiveRequest *handledBy = NULL;

   

    switch (errorCode)

    {

        // errors specific to Orders

        case 103:    // [from doc]  Duplicate order ID.

        case 104:    // [from doc]  Can't modify a filled order.

        case 105:    // [from doc]  request 45 error 105 Order being modified does not match original order

        case 106:    // [from doc]  Can't transmit order ID:

       

        …

        case 121:    // [from doc]  Invalid BD flag for the order. Check "Destination" and "BD" flag.

        case 122:    // [from doc]  No request tag has been found for order:

        // 123, 124 vague

        case 125:    // [from doc]  Buy price must be the same as the best asking price.

        case 126:    // [from doc]  Sell price must be the same as the best bidding price.

        case 129:    // [from doc]  VWAP orders must be submitted at least three minutes before the start time.

        case 131:    // [from doc]  The sweep-to-fill flag and display size are only valid for US stocks routed through SMART, and will be ignored.

        case 132:    // [from doc]  This order cannot be transmitted without a clearing account.

        case 133:    // [from doc]  Submit new order failed.

        case 134:    // [from doc]  Modify order failed.

        case 135:    // [from doc]  Can't find order with ID =

        case 136:    // [from doc]  This order cannot be cancelled.

        case 137:    // [from doc]  VWAP orders can only be cancelled up to three minutes before the start time.

        // 138, 129 vague

        case 140:    // [from doc]  The size value should be an integer:

        case 141:    // [from doc]  The price value should be a double:

        case 142:    // [from doc]  Institutional customer account does not have account info

        // 143 vague

        case 144:    // [from doc]  Order size does not match total share allocation.  To adjust the share allocation, right-click on the order and select “Modify > Share Allocation.”

        // 145 vague

        case 146:    // [from doc]  Invalid trigger method.

        …

        case 2102:    // [from doc]  Unable to modify this order as it is still being processed.

        case 2109:    // [from doc]  Order Event Warning: Attribute “Outside Regular Trading Hours” is ignored based on the order type and destination. PlaceOrder is now processed.

                    // actual:     Order Event Warning:Attempted modify of OutsideRth will not be done.  Order modify now being processed.

       

            // All the errors listed above are thought to be specific to Orders based on the message description.

            // It is sometimes unclear from the message whether the order id will be available with the error, or not.

            // Only errors with an id can be meaningfully routed to Order_error_Route.

            // All the rest might as well be routed right away to ReqTracker_error_Route.

            //

            // In any case the above list of errorCode's are treated as a list of what should be preferentially routed to orders before request trackers.

            // However the consequences of including an extraneous errorCode in the above list are non-existent for errors that turn out not to include a request id.

            // For errors with a request id, an extraneous errorCode in the above list is consequential only if it is NOT an order error,

            // but the request id coincidentally matches a currently-tracked order id.

            // It is hard to comment meaningfully on how obscure that is.

            // Instead it suggests motivation for a redesign that keeps order id's in their own numeric space,

            // or interleaves them in the same numeric space with other request id's.

            … rest of long comment omitted

            if (id != kIBNullReqId)

                handled = Order_error_Route (id, errorCode, errorString);

            else//10/14/12 was:  if (!handled)

                // So both kIBNullReqId errors and errors that are not successfully routed to an order based on the id will end up here.

                handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                    // see comment in default case re ReqTracker_error_Route details

            break;

            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

           

        case 326:    // error -1 326 Unable connect as the client id is already in use. Retry with a unique client id.

            clientIdInUse = true;

            break;

       

        case 505:    // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

            // Do not route this message–there should be no use in doing so since the connection is about to be closed.

            // Also see special case at top of this function.

            break;

       

        case 2100:    // actual:       New account data requested. API client has been unsubscribed from account data.

            handledBy = accountUpdateReqTracker .error_Route (id, errorCode, errorString);  // make sure AccountUpdateRequest::error_Generic gets this error

            AccountUpdatesCancelledByServer (errorString);

            break;

       

        default:

            handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                // calls ActiveRequestCategoryManager::error_Route which calls ActiveRequest::error_Route for each outstanding request

                // which calls error_SpecificRequest for request with matching id if found

                // and may call error_Generic which is currently only implemented by AccountUpdateRequest

                // for error 2100 "New account data requested. API client has been unsubscribed from account data."

           

            if (handledBy == NULL)

                if (id != kIBNullReqId)

                    (void) Order_error_Route (id, errorCode, errorString);

            break;

    }

   

    return handledBy;

}

and otherwise calls ReqTracker_error_Route which is also less than ideal but perfectly adequate and does this.

ActiveRequest *MyWrapper::ReqTracker_error_Route (const int id, const int errorCode, const CString& errorString)

{

    if (ActiveRequest *routedTo = historicalDataReqTracker  .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = marketDataReqTracker      .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = contractDetailsReqTracker .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = realTimeBarReqTracker     .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = accountUpdateReqTracker   .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = marketDepthReqTracker     .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = fundamentalDataReqTracker .error_Route (id, errorCode, errorString))

        return routedTo;

    if (ActiveRequest *routedTo = scannerReqTracker         .error_Route (id, errorCode, errorString))

        return routedTo;

    return NULL;

}

which in turn calls this

ActiveRequest *ActiveRequestCategoryManager::error_Route(const int id, const int errorCode, const CString& errorString)

{

    // Route all errors to all objects, until/unless an object's error_Route returns non-NULL, indicating it has "handled" the error.

    // In a typical implementation, generic errors (id == -1) will end up being routed to all objects,

    // and specific errors will be ignored except by the object with a matching request id (if any),

    // which will "handle" the error and return non-NULL (usually pointer to itself), preventing further looping here

    // (and in current implementation preventing equivalent of further looping in MyWrapper::ReqTracker_error_Route which calls this function for each request category until it is "handled").

    for (ActiveRequest *req = Next(); req != this; req = req->Next())

    {

        if (ActiveRequest *routedTo = req ->error_Route (id, errorCode, errorString))

            return routedTo;  // very likely routedTo will always equal req

    }

   

    return NULL;

}

and there the generic implementation of error_Route comes into the picture

ActiveRequest *ActiveRequest::error_Route(const int id, const int errorCode, const CString& errorString)

{

    if (this->ReqId() == id)

    {

        error_SpecificRequest (id, errorCode, errorString);

        return this;

    }

   

    if (id == kIBNullReqId)

    {

        if (errorCode == nonAPIErrorCode_connectionClosed)

            Aborted (errorCode, errorString);

        else

            error_Generic (errorCode, errorString);  // as of 5/3/12 this is the only call to error_Generic

    }

   

    return NULL;

}

And an implementation of error_SpecificRequest is provided by the request-tracking subclass for each API request type, such as this one.

void MarketDataRequest::error_SpecificRequest (const int id, const int errorCode, const CString& errorString)

{

    if (errorCode == 200)

    {

        Aborted (errorCode, errorString);

    }

   

    /* market data error reference

   

        error 39 200 No security definition has been found for the request

            the same error can happen for reqMktData and reqContractDetails and reqHistoricalData

            in this case it was a bogus combo (spread between AAPL and INTC options)

                sending MarketDataRequest 39 AAPL,INTC SPREAD

            but the "No security definition" error should be dispatched to the final request object

            and should maybe delete without cancel

           

        are there existing cases in which [??? text lost ???] caused an unintended cancel?

    */

}

So that may give you some sense for an approach.  While not ideal there is a lot of architecture there that supports quite a bit of experience working with the API.

-Kurt

[Q] Thoughts on “duplicate requestID” value (bad value given by API)

more details:

In my program, I get the NextValidId on connection to TWS and I use this

number to place my first order for the day, and I keep incrementing to place

the orders following it.

It so happened yesterday that the NextValidId returned by ID was a duplicate

one. The program had placed an order with order ID 638 few days back and It

placed the another order with the ID, 638.

(both the orders got executed)

I have unchecked the option "Reuse rejected order" in TWS — orders

configuration menu.

Does anyone have a clue why was an used OrderID returned by IB?

Thanks,

Vishruth

[A] by Frank Bell

I found it necessary to keep my own persistent copy of the current order

number. At start up I use max(my number, NextValidId).

Frank

[A] by Richard King

TWS stores the next valid id in its settings file. This is a large XML file,

and I suspect it's only written out when you shut down TWS (though I think

there is a save settings command in the menu? – can't check this at the

moment).

So if TWS crashed or you killed it rather than tidily closing it down, that

might account for your observation.

Richard

[A] by Jan Boonen

I also check the orderId of the EWrapper::openOrder( OrderId orderId, …) events, just in case of. If that order id is equal or bigger then the one to be used next, I set the next available value to orderId+1.

Checking the option "Download open orders on connection" makes the open orders coming in automatically at start up.

janB

[Q] Thoughts on “requestID space partitioned”

Hi Kurt,

Thank you for shading more light on the architectural constructs you developed.

Most definitely I'll reuse your approach of having one tracking instance for each individual request along with searchable lists (or maps as you suggested) to quickly retrieve corresponding objects for proper response routing/handling.

I was developing something very similar couple weeks before and part of the logic was automatic request ID generation at which point I recall that few months ago I read your take on request ID from some of the tread I can no longer easily find. If I recall correctly you said you track order request IDs separately from all other kinds of requests.

If it is the case, then my question is: how would you distinguish between "order's request IDs" space and "other request IDs" space if they overlapping? Example: you got error() call with ID matching one order tracking set and one from "other requests" set. Then route became ambiguous.

When I further thought about this problem I thought it would be generally good idea to add "expiration" field to each request, which can be set to something non-zero for requests with finite lifetime expectancy or to zero value for non-expirable ones (like for data subscription – should live "forever" until canceled). Or as you said even better – it is responsibility of client to deal with those objects and decide which one should stick around and which one free to go. But even hypothetical short life cycle of request tracking instances does not 100% complete less often "ambiguous route" cases if we try to separate orders from others. (and why do we want to do this in the 1st place).

ps: other possibility (if we need to separate orders from others by ID) would be to use odd numbers for orders and even for other requests, circling those spaces (like process ID assigned in linux OS:), but its ugly and unnecessary shrinks space for IDs twice (and I doubt we need same power of sets for orders vs other requests)

Thank you,

Dmitry

[A] by Kurt (from this thread: C++ architecture question)

[snip]

 

I was developing something very similar couple weeks before and part of the logic was automatic request ID generation at which point I recall that few months ago I read your take on request ID from some of the tread I can no longer easily find. If I recall correctly you said you track order request IDs separately from all other kinds of requests.

If it is the case, then my question is: how would you distinguish between "order's request IDs" space and "other request IDs" space if they overlapping? Example: you got error() call with ID matching one order tracking set and one from "other requests" set. Then route became ambiguous.

Mind you just because I have so far gotten away with this approach doesn't mean it is the best or that I will continue to get away with it.  I like to action on issues that are not pressing so I can keep considering different angles, or … maybe leave it as indefinitely.  Others have partitioned the requestID space.  I think Richard has done this.  It has been pointed out that it is a plentifully big space.

I just happen to like small numbers so I use small numbers for order IDs and small numbers for other request IDs.  Your odd/even idea (below) would also take care of this although I have the memory it may cause problems for certain kinds of orders for which IB may automatically insert another orderID on your behalf at the next integer value.

The reason I get away with it is that the error code space is somewhat defined by IB.  The MyWrapper::error_Route function which I mentioned but did not show has, among other things, a big switch on errorCode and a long list of errors known to be particular to orders, based on the IB documentation.  Here is that function with the long list of order-related errors truncated.

ActiveRequest *MyWrapper::error_Route(const int id, const int errorCode, const CString& errorString)

{

    if (errorCode == 505)

        // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

        printf ("generic error %d %s [id=%d]\n", errorCode, errorString.CStr(), id);

    else if (id == kIBNullReqId)

        printf ("generic error %d %s\n", errorCode, errorString.CStr());

    else

        printf ("request %d error %d %s\n", id, errorCode, errorString.CStr());

   

    bool handled = false;

    ActiveRequest *handledBy = NULL;

   

    switch (errorCode)

    {

        // errors specific to Orders

        case 103:    // [from doc]  Duplicate order ID.

        case 104:    // [from doc]  Can't modify a filled order.

        case 105:    // [from doc]  request 45 error 105 Order being modified does not match original order

        case 106:    // [from doc]  Can't transmit order ID:

       

        …

        case 121:    // [from doc]  Invalid BD flag for the order. Check "Destination" and "BD" flag.

        case 122:    // [from doc]  No request tag has been found for order:

        // 123, 124 vague

        case 125:    // [from doc]  Buy price must be the same as the best asking price.

        case 126:    // [from doc]  Sell price must be the same as the best bidding price.

        case 129:    // [from doc]  VWAP orders must be submitted at least three minutes before the start time.

        case 131:    // [from doc]  The sweep-to-fill flag and display size are only valid for US stocks routed through SMART, and will be ignored.

        case 132:    // [from doc]  This order cannot be transmitted without a clearing account.

        case 133:    // [from doc]  Submit new order failed.

        case 134:    // [from doc]  Modify order failed.

        case 135:    // [from doc]  Can't find order with ID =

        case 136:    // [from doc]  This order cannot be cancelled.

        case 137:    // [from doc]  VWAP orders can only be cancelled up to three minutes before the start time.

        // 138, 129 vague

        case 140:    // [from doc]  The size value should be an integer:

        case 141:    // [from doc]  The price value should be a double:

        case 142:    // [from doc]  Institutional customer account does not have account info

        // 143 vague

        case 144:    // [from doc]  Order size does not match total share allocation.  To adjust the share allocation, right-click on the order and select “Modify > Share Allocation.”

        // 145 vague

        case 146:    // [from doc]  Invalid trigger method.

        …

        case 2102:    // [from doc]  Unable to modify this order as it is still being processed.

        case 2109:    // [from doc]  Order Event Warning: Attribute “Outside Regular Trading Hours” is ignored based on the order type and destination. PlaceOrder is now processed.

                    // actual:     Order Event Warning:Attempted modify of OutsideRth will not be done.  Order modify now being processed.

       

            // All the errors listed above are thought to be specific to Orders based on the message description.

            // It is sometimes unclear from the message whether the order id will be available with the error, or not.

            // Only errors with an id can be meaningfully routed to Order_error_Route.

            // All the rest might as well be routed right away to ReqTracker_error_Route.

            //

            // In any case the above list of errorCode's are treated as a list of what should be preferentially routed to orders before request trackers.

            // However the consequences of including an extraneous errorCode in the above list are non-existent for errors that turn out not to include a request id.

            // For errors with a request id, an extraneous errorCode in the above list is consequential only if it is NOT an order error,

            // but the request id coincidentally matches a currently-tracked order id.

            // It is hard to comment meaningfully on how obscure that is.

            // Instead it suggests motivation for a redesign that keeps order id's in their own numeric space,

            // or interleaves them in the same numeric space with other request id's.

            … rest of long comment omitted

            if (id != kIBNullReqId)

                handled = Order_error_Route (id, errorCode, errorString);

            else//10/14/12 was:  if (!handled)

                // So both kIBNullReqId errors and errors that are not successfully routed to an order based on the id will end up here.

                handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                    // see comment in default case re ReqTracker_error_Route details

            break;

            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

           

        case 326:    // error -1 326 Unable connect as the client id is already in use. Retry with a unique client id.

            clientIdInUse = true;

            break;

       

        case 505:    // actual:    "request 2104 error 505 Fatal Error: Unknown message id."

            // Do not route this message–there should be no use in doing so since the connection is about to be closed.

            // Also see special case at top of this function.

            break;

       

        case 2100:    // actual:       New account data requested. API client has been unsubscribed from account data.

            handledBy = accountUpdateReqTracker .error_Route (id, errorCode, errorString);  // make sure AccountUpdateRequest::error_Generic gets this error

            AccountUpdatesCancelledByServer (errorString);

            break;

       

        default:

            handledBy = ReqTracker_error_Route (id, errorCode, errorString);

                // calls ActiveRequestCategoryManager::error_Route which calls ActiveRequest::error_Route for each outstanding request

                // which calls error_SpecificRequest for request with matching id if found

                // and may call error_Generic which is currently only implemented by AccountUpdateRequest

                // for error 2100 "New account data requested. API client has been unsubscribed from account data."

           

            if (handledBy == NULL)

                if (id != kIBNullReqId)

                    (void) Order_error_Route (id, errorCode, errorString);

            break;

    }

   

    return handledBy;

}

So this code errs on the side of NOT routing errors to orders first, except in the case when the error code is specifically known to deal with orders.  The worst consequence of this is not really SO bad because the order callback messages will take care of the basic state of things most of the time, and if an unexpected state change occurs, there is at worst probably a message in the log to explain why, at which point the routing code can be modified as appropriate.

I'm not necessarily advocating this approach, but it is the one I took.

[A] by Richard L King

Regarding partitioning of the ID space, I use the following ranges for

various request types.

Market data requests: 0 – 0x3FFFFF

Market depth requests: 0x400000 – 0x7FFFFF

Historical data requests: 0x800000 – 0x80FFFF

Execution details requests: 0x810000 – 0xFFFFFF

Contract details requests: 0x1000000 – 0xFFFFFFF

Order ids: 0x10000000 – 0x7FFFFFFF

Partitioning the ids in this way makes it easy to identify what type of

request an error message refers to, indeed the error can be routed directly

to the handler for that sort of request.

There is no particular significance in the size or placement of these

ranges, apart from the fact that they are all plenty big enough for all

practical purposes (no doubt someone will prove me wrong by running their

trading client for 514.8 years at 10000 orders per day, but I may not care

by then).

In my API implementation, clients make their requests using ids in a range

from 0 to n, where n depends on the type of request and is derived from the

id ranges . Thus market data request ids are in the range 0 – 4194303,

execution details requests 0 – 8323071. This approach allows clients to use

the request id as an index into an array to locate the relevant handler

object for the response, rather than searching a list or using a hash lookup

(the time saving is really irrelevant, but this sort of thing appeals to my

vanity).

When a request is made, the API code translates the caller's request id into

TWS's id space by adding the appropriate base value, and vice versa when

passing out the response.

Order ids are handled slightly differently in that the client plays no part

in allocating them. The placeOrder method itself allocates the id to the

supplied order object. Note that if the supplied order already has a

non-zero id then this is just an order modification and no id needs to be

allocated. I considered having an explicit modifyOrder method, but decided

against it.

Richard

[Q] Trying to modify an order and sometimes FAIL!?

[A] by Kurt (taken from this thread)

That may be a solution, but you haven't addressed the problem. Here is what

*might* be your problem. Since it is the "standard" symptom that I'm so

familiar with I think it is likely. You would have this problem if you save

an order returned by the openOrder message and use it for subsequent calls

to placeOrder.

It is a bug in processMsg. In C++ the bug is in

EClientSocketBase::processMsg, case OPEN_ORDER.

With every new release of the TWS API I go through it and make the same

fixes. See the following block of 5 lines with my changes marked in two

places:

DECODE_FIELD_MAX/*kkb–was DECODE_FIELD*/( order.volatility); // ver 11

field

DECODE_FIELD( order.volatilityType); // ver 11 field

DECODE_FIELD( order.deltaNeutralOrderType); // ver 11 field (had a hack

for ver 11)

DECODE_FIELD_MAX/*kkb–was DECODE_FIELD*/( order.deltaNeutralAuxPrice);

// ver 12 field

DECODE_FIELD( order.continuousUpdate); // ver 11 field

The problem is that IB's code is defaulting absent order.volatility and

deltaNeutralAuxPrice fields incorrectly when the fields are missing (empty

string) from the incoming OPEN_ORDER message. DECODE_FIELD_MAX rather than

DECODE_FIELD is needed to detect the empty string case and substitute the

appropriate "UNSET" value, in this case UNSET_DOUBLE (which is defined as

DBL_MAX, and thus the name DECODE_FIELD_MAX).

The DECODE_FIELD versus DECODE_FIELD_MAX should probably agree with the

default values set by the Order constructor and in these two cases they do

not, and it causes a problem. Scan the Order constructor for how numeric

fields are initialized. Some are initialized to zero and some are

initialized to UNSET_DOUBLE or UNSET_INTEGER as appropriate for the type.

processMsg should probably be using the UNSET value for missing fields. In

short missing message fields should get the default value that the Order

constructor would have given them. Or I believe that is a reasonable model

anyway.

As a result of this problem, the openOrder message returns a corrupt Order

to you. If you use it with placeOrder it will fail, because placeOrder

wants to see the value UNSET_DOUBLE in fields that are not applicable for

non-volatility orders. Instead openOrder is putting a zero in these fields.

Make the two fixes above and the problem goes away. These are the only two

corrections I make to the EClientSocket stuff. I believe there may be other

places where processMsg disagrees with the Order constructor, and I have

them noted somewhere, but they have not caused me any actual problems, so I

don't bother fixing them on every API release.

-Kurt

[Q] Trying to modify trailing stop, got an error 10067

[A] by Josh View/Reply Online (#39111)

Added on 13-Dec-2017

So it looks like the order didn't immediately trigger and a trail stop price was not assigned at 19:25:02. Therefore it could be modified without specifying a trail stop price. Later, there was a trade in the market which triggered the system to modify the order with a trailing stop price. When an attempt was made to modify the order again (19:25:06), it no longer matched the order in the system because the trailing stop price had changed. To modify the order at that point it would be necessary to use the Order object in the most recent orderStatus() message returned instead of the initial orderStatus which did not show a trailing stop price.

[Q] Order execution messages considerations

[A] by Kurt (from here)

I could be missing something re the "multiple" messages, but will hazard a

guess.

You should never get two execution messages with the same execId. You have

to keep track of such things if you want any reliability. Duplicate

messages should not be a problem whether they are orders or executions as

long as you maintain the necessary information. (Well I can't speak for

what is possible within Excel.)

Some people think it is trouble to keep track of a state for each object

they are concerned with. But the trouble of not doing so is immense. You

create problems that don't really exist. IB is not trying to create the

kind of system that would be reliable to use without keeping track of state

at the other end. So if you don't maintain the necessary information, the

problems you get as a result are essentially problems you created.

You can request executions at any time, and then you will get more

"duplicates", but again just ignore the ones you already received. As I

recall some people request executions explicitly because they are seeing

execution messages getting dropped. (This is a reason to track both orders

and executions since conceivably either might get dropped.)

-Kurt

[Q] Checking API connection / checking the timeliness of ticks

[A1] by rwk2095

> I subscribe to an issue (e.g. ES or QQQQ) that I know should be

> updating.I timestamp all updates including bids and offers,

> and raise an audible alarm if I haven't received an update within

> some time interval such as 10 seconds.

I like the idea, thanks. It leaves out stock specific problems but catching all connectivity and market-specific problems with one little sub is nice.

[A2] by Mike Smith

For entering positions my ATS watches a fast moving average of volume and I consider it a "bad thing" if volume is accelerating or decelerating beyond a certain point (as confirmed by back testing) and I do not make the trade. As a side effect that takes care of cases where ticks stop coming in for whatever reason.

Now for very slow markets you are describing you will have to back test it but you might find the same thing – that a (very) long pause of little or no volume is not the ideal entry point, so the same technique might work. But I would do two thing: (1) Do not use SMART as the exchange, use the highest volume exchange for the market you trade, for stocks that's usually NYSE or highest volume ECN for that symbol. (2) Find a high volume symbol using the same exchange and reqMarketData for that symbol as well but just throw away the data and look for pauses. But then it is as Jim say, what are you going to do? I have a notification system (text msg and email) that kicks in when anything gets too wacky. There are some cases where you just have to be there and make a "human" decision, long pauses in ticks being one of them.

Mike

[Q] AvgPrice rounded to 2 digits, I want 3 digits!-)

[A] read this thread

[Q] How to get avgFillPrice for multiple filled order status updates?

[A] by alex_7880

orderStatus() also returns "int remaining". My logic is basically to wait for remaining to be 0 before I consider the order to be filled, and I use the "double avgFillPrice" that is returned when remaining=0.

[Q] Rounding to the tick size of the Financial Instrument

[A] by Mike Smith from this thread

You can automate figuring out the number of decimal places. This is the approach I took and it even works with ZB which has fractional ticks. I have a SymbolInfo class which contains a nearestTick() method that does what you want:

class SymbolInfo {

// This is assigned a value in constructor (below)

private int tickSize;

////////////////////////////////////////////////////////////

// Note: These two class vars are used only by nearestTick()

//

// The value of nearestTickMultiplier is based on the number

// of digits in the fractional part of the tick size. It's

// 10 to the power of that many digits. So if the tick size

// is a penny (0.01) then the number of digits in the

// fractional part is 2 and nearestTickMultiplier is 100.

private long nearestTickMultiplier;

//

// The value of tickSizeMultiplied is the tick size multiplied

// by nearestTickMultiplier. It is used to calculate the price

// as a long int which is the smallest possible long int which

// can be used to calculate the price by converting it to a

// double and dividing by nearestTickMultiplier. This guarantees

// price is a power of 10 without roundoff errors.

private long tickSizeMultiplied;

//

// Another example would be for futures contracts which use 1/32

// as a tick size which is 0.03125. In this case the number of

// digits in the fractional part is 5 so nearestTickMultiplier

// would be 100000 and tickSizeMultiplied would be 3125.

//

// This would also come in handy for representing tick values of

// FOREX symbols where one currency relative to the Dollar might

// be something in the neighborhood of 0.000000000001 to 1 (or

// visa-versa the way things are going lately).

///////////////////////////////////////////////////////////////

// Constructor

public SymbolInfo(ContractDetails contractDetails) {

tickSize = contractDetails.m_minTick;

int n = getFractionDigitCount();

// nearestTickMultiplier is always 1 or greater.

nearestTickMultiplier = Math.round(Math.pow(10.0, n));

tickSizeMultiplied = Math.round(tickSize * nearestTickMultiplier);

}

/**

* Get number of digits in fractional part of tickSize

*

* @return Number of digits in fractional part of tickSize

*/

public final int getFractionDigitCount() {

double fraction = tickSize – ((long)tickSize);

String s = Double.toString(fraction);

int n = s.length();

if (s.contains(".")) {

if (s.equals("0.1")) {

n++; // force trailing zeros, just looks better (GC)

} else {

while (s.charAt(n-1) == '0') {

n–;

}

}

}

if (s.startsWith("0")) {

n -= 1; // remove leading zero

}

n–; // remove decimal point

return n;

}

/**

* Round given price to nearest tick boundary so

* the returned value is a multiple of the tick size

* for this instrument.

*

* @param price The price we want to adjust

* @return The price adjusted to the nearest tick boundary

*/

public final double nearestTick(double price) {

long ticks = Math.round(price/tickSize);

price = (double)(ticks * tickSizeMultiplied);

return price/nearestTickMultiplier;

}

}

[Q] Should I use MKT or LIMIT orders? How do I track partial fills?

[A] by tdrtw from this thread

Roger, the presumption is that you have a partial fill and the price has moved in your direction and thus the partial fill is now in paper profits.

When I manually traded I used to enter at an exit price using a STOP-LIMIT order to specify that price and often had a partial fill in a fast moving market. Sometimes I waited and the price came back and the fill completed.

When I moved to a fully automatic system, I changed my rules and decided to Enter on a STOP order with no Limit Price.

Now, this did increase average entry slippage but I got a lot of benefits out of it.

1) So far every entry order has always filled.

2) Now that the entry orders are filled, exit orders are placed within a split second. Which is much safer than a partial fill open out there and having complex software rules to manage this.

3) I now catch fast moving markets and take profits on orders that once I never got into using Stop Limit entries.

4) Yes once in a while the slippage does go against me.

Overall, I have found no detriment to system performance. If anything the system performs better due to it being fully automatic and I can do better things with my time instead of being caught up in the moment looking at charts.

My system is much safer in automode using a STOP entry than STOP-Limit entry with exact price.

I know it can be hard, but sometimes it's better to modify your system to fit software, than create a complex software program to force it to run your current rules.

Since being automatic I now create much simpler systems that are easy to program and less chance for things to go wrong.

That's just my opinion.

Good luck!

Damien

(aka Exotric_Adriana_1982 😉

[Q] Why PosixMQ is a good idea for IPC? Why can’t I just use some thread-safe queue (with mutex)?

[A] by Nils Pipenbrinck from stackoverflow

In realtime OS environments you often face the problem that you have to guarantee execution of code at a fixed schedule. E.g. you may have a function that gets called exactly each 10 milliseconds. Not earlier, not later.

To guarantee such hard timing constraints you have to write code that must not block the time critical code under any circumstances.

The posix thread synchronization primitives from cannot be used here. You must never lock a mutex or aqurie a semaphore from time critical code because a different process/thread may already have it locked. However, often you are allowed to unblock some other thread from time critical code (e.g. releasing a semaphore is okay).

In such environments message queues are a nice choice to exchange data because they offer a clean way to pass data from one thread to another without ever blocking.

Using queues to just set variables may sound like overkill, but it is very good software design. If you do it that way you have a well-defined interface to your time critical code.

Also it helps to write deterministic code because you'll never run into the problem of race-conditions. If you set variables via message-queues you can be sure that the time critical code sees the messages in the same order as they have been sent. When mixing direct memory access and messages you can't guarantee this.

[A2] by Alok Save from stackoverflow

Message Queues are predominantly used as an IPC Mechanism, whenever there needs to be exchange of data between two different processes. However, sometimes Message Queues are also used for thread context switching. For eg:

You register some callback with a software layer which sits on top of driver. The callback is returned to you in the context of the driver. It is a thread spawned by the driver. Now you cannot hog this thread of driver by doing a lot of processing in it. So one may add the data returned in callback in a message Queue, which has application threads blocked on it for performing the processing on the data.

[some feedback from 100% complete frameworks out there;-]

[A] by poch32

Depending on the security, my ATS currently checks for anything between 3 and 8 stratregies. Strategies are given a "privilege ring" (so to speak in OS jargon) thus, in case of clashing, one will always prevail among the others.

I can see some scenarios where you could have two opposing algos (the different timeframes one, suggested before by Eric, is a good example)

However, in my case, since all my strategies are – more or less – oriented towards the same timeframe, I cannot allow 2+ strategies to collide, since that would be, effectively, giving money away in comms.

If I had two independent, concurrent instances of my ATS running, and I was in the need of allow concurrent strategies to collide, I would clearly go down the pipe/tcpip route. Preferably the last one, since sooner or later perhaps I could need to move one of the ATS to another machine.

[A] by tdrtw

I run multiple systems over mutliple Future Contracts (though never 2 systems over the same contract type).

I created a shell java console app that runs on a Windows XP Pro machine and communicates with an Oracle XE database.

The shell java console app can ONLY trade 1 financial instrument (FI from here on). It is an input paramter when I start the .jar file which maps to a database control table to get all the info such as expiry, how much to risk on the trade, tick size, strategy type etc..

The database control table also tells it what strategy to trade. The java console app makes a generic call to an Oracle Stored Procedure that will then call the correct strategy which is just coded in an Oracle Stored Procedure.

Basically when I boot up each day, I start multiple instances of the java console app each pointing to a different FI which each trades it own strategy.

Took me a long time to get it to work, as I went down the wrong path many times. But one thing I can say, it works f&*ing wonderfully, and as you can tell, I'm a little proud of it 🙂

I had prior skills working on an Oracle database, but never had any skills in Java when I started the automation venture many years ago. So I leaned on my Oracle Database skills and have the code split between java and oracle pl/sql.

Now that I look at it, it's no big deal but it was a little challenging for me at the beginning. Stick to it and it will fall in place. Better to go 10 steps forward and 5 steps back than just stand still.

Also be careful of leaning too much on these boards. I remember once sitting next to an experience person and found myself constantly asking them questions which I could really work out the answers to. Once that person moved, I found myself working things out for myself more and actually suprised myself in a sense that I can actually do it if left to sink or swim.

Good Luck, you will find your path eventually!

Damien

[A] by Richard L King

Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.

2. Develop a completely separate daemon process that runs one or more instances of IBController based on configuration data: eg run IBController on the live account at 06:00 on Monday mornings, stop it (via IBController’s command channel) at 22:00 on Friday, and if the IBController process drops out at any point between these two times then restart it. Note that there is already sample code for doing this on Windows in the IBControllerService sample (though less sophisticated than what I just described).

Actually I already do something very similar to 2. using nothing more than the Windows Task Scheduler. So if for example there’s a power failure during the week, or IBController/TWS crashes (though it never does), IBController is automatically restarted when power is resumed, but not for a weekend failure.

What I don’t do is any kind of active monitoring of TWS or the API (by poking it in some way), because during 11 years of running API programs all day every trading day, I’ve never found any need for it. (Your mileage may vary, though I don’t see why. But maybe if I were pushing hundreds of order per hour through the API, or continually downloading historical data, it would be a different matter.)

I also don’t see any need for trading systems to wait until IBController is ‘ready’ in some sense. My platform doesn’t care in the slightest whether IBController or TWS is running – if so configured, it simply keeps periodically retrying the connect until it succeeds. Indeed I run most API programs on different machines than the one running IBController/TWS/Gateway.


[Q] Unit testing – which one to choose?

[A] by Dmitry (added on 07-May-2015)

Very good article on the C++ Unit Testing

“Exploring the C++ Unit Testing Framework Jungle”

http://gamesfromwithin.com/exploring-the-c-unit-testing-framework-jungle

also see:

http://stackoverflow.com/questions/1407354/is-there-a-c-unit-testing-library-that-is-similar-to-nunit

good wiki on unit testing comparison (it has other languages as well!)

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks


[other chaotic implementation notes]

———————————————————————————————————–

Tick momentum indicator

An Idea by MarketMole (from this thread)

I've also built tick momentum indicators which do indeed try to determine the side that trade fell on as far as the going bid/ask spread was concerned. I know I've run these against various brokers but I'm sure I also ran against IB at some time and (thought) I was able to determine general trade sides, in a broad sense.

———————————————————————————————————–

On topic: “What to monitor [remotely]”

by Jason Sapp from this thread

I do not monitor actual positions through my GUI, but rather just watch my Net P/L, Worst Case Situation, Unrealized P/L, Realized P/L, Position Count (long/short), Missed Positions, etc….  I've thought about putting in functionality to monitor each position, but I just haven't done it.  I also want to eventually implement the capability to close out positions manually (if need be), but (once again), just haven't had to real need to do so.

I've attached a screen shot so that you can see what it looks like (yesterday just happened to be a good day for me, so my real-time graph looks quite nice).  Please excuse some of the anomalies towards the end of my yellow line (on the attached graph), where my Worst-Case exceeds my Net P/L.  It's just the way I'm sometimes capturing data at the moment a position is being closed.  I just haven't taken the time to fix the bug.

3h2dg.jpg

Note: this was drawn using http://www.amcharts.com/demos/

———————————————————————————————————–

It's not intuitive but IB only sends the deltas of price and size, not

complete tick events.

If a trade occurs at the same price as the previous price message, you

will only get a size message (the price is implicitly the previous price

message).

———————————————————————————————————–

rwk:  I only use order status events for logging

zsolt: Final q, it is worth to correlate OrderStatus and Exections data, and make

an alert if they don't agree, in order to confirm that TWS and/or ATS works

correctly, or, Executions data is reliable in practice to trust?

———————————————————————————————————–

I have never missed an execution (WHILE CONNECTED), but others here

have. Although *I* have not missed an execution, others here

have, so IB isn't 100% reliable, just using execDetails.

You should write the reconciliation code, and the replay code,

because you WILL need it if your connection breaks.

———————————————————————————————————–

Nice, thanks. Last week i have created the reconciliation code (even I named

my class as OrderReconciler :), which compares my persisted state in db with

openorders and executions data, and push executions and cancellations to the

alg to update their state in memory and in db.

In this way I can restart the app without losing context, but naturally i

might miss some trading opportunity.

Until now, this seems to work correctly.

Thanks for the suggestions for everybody about execdetails.

———————————————————————————————————–

That's right re updatePortfolio. And your scenario is the same reason I

don't use rwk's approach. However if you are reconciling things against

each other especially *during* trading, updatePortfolio is potentially damn

useful since if any change occurs in updateportfolio that is not predicted

by the order/execution messages you have received, then you know you

probably missed something. Under that condition your code could request

open orders and/or request executions again, and/or produce an alarm.

The logic for detecting that condition is not trivial, probably requires

some time slop, could possibly consider all possible combinations of orders

that *might* have executed or partially executed, etc.

-Kurt

———————————————————————————————————–

[good implementation note on “designing stop-limit orders”]

found here: src

a possible solution:

use both stop order and stop limit order with OCO

for example, you want to buy stop at 10.00

so you put a stop-limit order at 10.00 and a buy stop order at 10.10

let them OCO to ensure only one order will be triggered

———————————————————————————————————–

Notes on ContractTracker implementation

I've created ContracTracker class in "speculant", which handles 3-5 IB API requests under the hood to provide the framework with some history plus real-time data (price ticks, realtime bars).

Here's some high-level details:

Basically IB provides 5 ways to get prices for a given contract:

  1) reqMktData gives you stream up sampled price updates (say you get 3-5 price values per second, but can miss high/low)

  2) reqHistoricalData (with argument "keep_up_to_date" set to true) – this gives you bars from history up to now and then it will continue sending you bar updates (again 3-5 times / sec, but not just 1 price value, but whole curren bar with updated OHLC values) – I forgot the exact limit, but I'd guess you can't have >50 "realtime" requests running same time, so if you use "3)" and "1)" as well – those all will count, thus if you want to have all 3 types of "real time" updates per each contract in your platform, then real limit will be around 50/3 = 16 contracts simultaneously, no more : (

  3) reqRealTimeBars – this one gives you 5sec bars update sent to you only (one update every 5 second)

  4) then there is subscription to real ticks with 1000s of updates, but you only limited to 3 contract subscription

  5) then there is "depth" where you also get all the bids and asks (level-2)

My goal is to be able to track up to 50 realtime contracts.

I use only 1st three and I put a class (java), which asks for some history, but with no "keep up to date" set to true (I ask only for history), then I ask for 5-sec realtime bars and for sampled prices updates (reqMktData).

Then I "merge" them and use all incoming updates to build my own 1-min bars, "improved" 5-sec bars (instead of 1 update every 5 sec, my bars got updated on each tick price update few times/sec) and also I build synthetic 1-day bars.

———————————————————————————————————–


Well known errors and how to avoid them

[Q] List of documented Error Codes?

[A] by IB API 🙂

Always start searching the error code in the official TWS API documentation:

https://interactivebrokers.github.io/tws-api/message_codes.html

[Q] reqSecDefOptParms() with exchange specified returns no data?

I find that reqSecDefOptParms() with a non-empty exchange parameter never returns any data. But if the exchange parameter is blank, it returns data for all exchanges.

 

For example:

 

reqSecDefOptParams(0,"MSFT", "STK", "CBOE", 272093)

 

gives nothing whereas:

 

reqSecDefOptParams(0,"MSFT", "STK", "", 272093)

 

returns data for AMEX, BOX, CBOE2, BATS, ETC.

 

Am I doing something wrong, or is it just broken?

 

Richard

[A] by Josh View/Reply Online (#44803)

Added on 27-Jun-2020

The exchange field is only for futures/futures options, it should be labelled 'FutFopExchange'. For stocks it should be left blank.

Josh (IBKR)

—- also later: summary from Richard —

So to summarise what is still a very illogical situation regarding the futFopExchange parameter of reqSecDefOptParams():

 

  • The documentation says:

"futFopExchange The exchange on which the returned options are trading. Can be set to the empty string "" for all exchanges".

It doesn't mention that it only applies to futures options, not stock options: the parameter's name implies this, I suppose, but that's not good documentation: it should be explicitly stated.

  • If you DO set it to the empty string for futures options, you get "Error 321 … Missing exchange for security type FUT".
  • Because futures options only appear to trade on the same single exchange as the future itself, there is only one possible choice for this parameter for any given underlying. So the parameter is completely useless.
  • If you set it to the empty string for stock options, you get info for all exchanges returned, as expected.
  • If you DON'T set it to the empty string for stock options, you get nothing at all – not even an error!

 

So can anyone see any logic in this? The futFopExchange parameter accomplishes precisely nothing, except cause trouble.

 

I can't help thinking I'm missing something, but I've no idea what…

[Q] I observe gaps in the most recent historical date (options in my case). But later same data looks good. Why?

[A] by Jeff View/Reply Online (#40888)

Jeff, Unfortunately you may be encountering an issue where there is a lag in recording the most recent historical options data to the server database. For tracking status updates it would be best to contact customer service. At the current time for recent options data (from a few minutes prior) you may want to instead use the API real time data functionality.

[Q] Errorcode(110): The price does not conform to the minimum price variation for this contract

[A1] read this thread (it’s only 43 messages;)

[A2] by Nick View/Reply Online (#38922)

The message means that the limit price is not a multiple of the minimum price increment of the stock. For example, if the stock trades in increments of one cent a limit price of 20.153 would not be valid. This is usually due to a price that comes from a calculation, like a moving average.

[Q] Errorcode(162): Error message:No market data permissions (for some tickers, even with proper subscription!)

View/Reply Online (#43032)

I can't download historical data for certain tickers.

Error 162, reqId 9: Historical Market Data Service error message:No market data permissions for AMEX, ARCA, BATS, BEX, BYX, DRCTEDGE, EDGEA, ISLAND, NYSENAT, PSX STK, contract: Stock(symbol='sfet', exchange='SMART', currency='USD')

SFET is foreign company but it is listed on NASDAQ, same happened with IMRN yesteday (also listed on NASDAQ). I don't think i had problem like this before.

I'm subscribed to classic level 1 data. It should work when it's listed on NASDAQ.

[A] by toth.david6@gmail.com View/Reply Online (#43036) 

Added on 04-Oct-2019

Problem solved. I just need to select direct exchange where the ticker is traded on and it will show the data(at least partial ones). These tickers are traded on more exchanges and if i choose SMART and i am not subscribed to every exchange its traded on i will get the error via API or delayed data in TWS platform.

I thought it's little smarter and it will give me data from all the exchanges i am subscribed to and aggregate it accordingly, but if youre missing just one exchange it will give you error right away. That $4.5 level 1 data bundle cover doesn't cover everything in this case.  

[Q] Errorcode(162) Historical Market Data Service error message:HMDS query returned no data: GCQ9@NYMEX Trades

[A] by Josh View/Reply Online (#43045)

Added on 08-Oct-2019

That error most commonly occurs when data is requested outside the date range when the product was trading. When I try it for 7/1/19 for instance it returns data ok.

[Q] Errorcode(162) 'Historical Market Data Service error message:Starting time must occur before ending time'?

I'm downloading 1 second historical bars and got almost a year of data (2000 bars per one request, 10 seconds between requests), but at some point it starts to return mentioned above error, which makes no sense, since in the request there's only 1 "end date" argument, there is no "starting time" whatsoever.

[A] by Dmitry

Added on 08-Mar-2020  Happy Women's Day!  🙂

Consider this as "no more data available". IB can also return same error number 162 with text "HMDS query returned no data", which means the same. If you need more years of history, then consider asking larger bars (instead of 1 secs bar try 1 min, 10 min etc.)

[Q] Errorcode(162) 'Historical Market Data Service error message: Scanner filter marketCapAbove is disabled'?

[A] View/Reply Online (#46306)

Added on 27-Jan-2021

Not really an answer, but 1st time ever error mentioned when using scanners.

[Q] Errorcode(200) No security definition has been found for the request

by enthalpyyan@gmail.com

Hi Guys,

When I place a option order with the following arguments, I always got 200 error.

        Contract contract;

        Order order;

        contract.localSymbol = "MSFT131206P00034500";

        contract.secType = "OPT";

        contract.currency = "USD";

        contract.exchange = "SMART";

        order.action = "BUY";

        order.totalQuantity = 1000;

        order.orderType = "LMT";

        order.lmtPrice = 20.00;

        m_state = ST_PLACEORDER_ACK;

        m_pClient->placeOrder(m_orderId, contract, order);

How can I solve this problem? Thanks.

[A1] by souqmate@yahoo.com

make sure you use the result of reqContractDetails() on

symbol="MSFT" secType="OPT" exchange="SMART" currency="USD"

In your case, you'd rather use localSymbol="MSFT  131206P00034500".

Specifying symbol="MSFT" is good practice, but doesn't seem necessary in this context (at least for reqHistData).

See also: [Q] futures options data via API

souqMate

ALSO

you don't need primaryExchange = "NASDAQ",

but you need expiry = "20131213"  (YYYYMMDD), as that's the output of reqContractDetails.

play around.

souqMate

[owner reply]

Hi, thanks.. placeOrder() works now with the following:

  contract1.secType = "OPT";

  contract1.multiplier = "100";

  contract1.exchange = "SMART";

  contract1.currency = "USD";

  contract1.includeExpired = 0;

  contract1.localSymbol = "";

  contract1.right = "Call";

  contract1.symbol = "MSFT";

  contract1.expiry = "20131206";

  contract1.strike = 36.5;

        order1.action = "BUY";

        order1.totalQuantity = 1;

        order1.orderType = "LMT";

        order1.lmtPrice = 0.2;

But I get stucked again.. I want to place a combination option order, how can I do this in C++? IB's API doc really sucks and I can't find useful infomations..

——- AND LATER: ——-

IB's API really makes me crazy. With the same settings, I can place an order with MSFT but I can't place with like AAPL or GOOG. It is the same error "no security definition has been found for the request". I am totally confused about how can I place an order.

[A3] by souqmate@yahoo.com

Before you try to place an order, make sure you have output from reqMktData (or reqHistData when markets are closed).  For that, the output of reqContractDetails helps in specifying the contract; eg get 628 contracts for symbol="MSFT" secType="OPT" exchange="SMART" currency="USD".

There might be factor 100 hiding in strikes (UK stocks), etc.

Also, beware that some functions misbehave on the demo account.

Keep us posted once you succeed in your combo.

souqMate.

[Q] Errorcode(201): Order rejected – reason:It does not comply with IB's order handling rules for derivatives that result in physical delivery.

[A] by Kurt

So it may not turn out to be an API question.  At least it seems evident that the problems you describe relate to the actual differences between the accounts themselves.  I'd suggest the next step is probably to do things manually in TWS and then call IB for support as needed, to clarify account configurations and distinctions so you can know what you are working with.

-Kurt

[Q] Errorcode(300): Can't find EId with tickerId:40000001.

Added on 08-Nov-2017

[A] by Josh View/Reply Online (#38925)

John,

The error "Can't find EId" is typically triggered when an attempt is made to cancel a subscription that was already cancelled or with an incorrect tickerID. I'm not in front of a Windows computer but trying the Java API here it works ok. Are you using the edemo account by chance? The real time bars have never worked there, one of the problems of testing with that account.

Josh

[Q] Errorcode(321): Missing exchange for security type FUT

[A] see: [Q] reqSecDefOptParms() with exchange specified returns no data?

Added on 27-Jun-2020

[Q] Errorcode(321): Error validating request:-'rc' : cause – You must specify an account.

[A] by Kurt

The account (or perhaps called m_account) field of the Order must be set if you have an FA login.  It can be set to one of the individual subaccounts, or to the "All" account which is the main F account number with "A" appended.

Also, when I last checked, the account field must be set on the first call to placeOrder for a given order.  It can not be set on a modification.

I think someone said that groups and allocations can be made to work also, but I believe this used to not work in the situation I needed for some reason, so I have not tried it again and am not familiar or up to date on this.

-Kurt

[Q] ErrorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – Historical data queries on this contract requesting any data earlier than 199 year(s) back from now which is 18210405 00:18:14 GMT are rejected.  Your query would have run from 5240807 23:29:46 UTC to 5240807 23:33:06 UTC.'

[A] Well obviously reqHistoricalData is not happy with endDateTime argument value, check it carefully.

[Q] errorCode: '321', errorMsg: 'Error validating request:-'bS' : cause – End date not supported with live updates'

[A] This might happen when you used keepUpToDate=true in your reqHistoricalData() call. I found a solution here. Quote: I have mail to ask the IB team. He told me to leave endDateTime as blank. It's work!!

[Q] errorCode: '322', errorMsg: 'Error processing request.Only 50 simultaneous API historical data requests allowed.'

[A] by Dmitry. Learned the hard way: reqHistoricalData with keep_up_to_date set to true produces on average bar updates every couple seconds, but you can only have 50 such requests running simultaneously. Subscribing to what_to_show=BID and what_to_show=ASK are 2 different subscriptions, so if you want to subscribe to real time BID, ASK and TRADE, then 50/3 = 16 contracts max.

reqRealTimeBars on the other hand would allow to have 100 subscriptions running simultaneously, but it only gives you updates strictly once per 5 seconds.

Btw, if you need bid/ask, consider using what_to_show "BID_ASK" (see Historical Data Types).

[Q] errorCode: '456', errorMsg: 'Max number of real time requests has been reached.'

[A] similar to errorCode: '322' (see above). You seems to have too many reqRealTimeBars and/or reqMktData requests going on simultaneously (more than 100).

[Q] Error 478 (undocumented): Parameters in request conflicts with contract parameters received by contract id: requested strike 1300.0, in contract 13.0;

[A]  Options and future options priced in cents/pence instead of USD/GBP

[Q] Error 481, Order size was reduced

I get a lot of those on my FA account.

Anybody figured what this error means and how to avoid it?

Funny thing that I don't see any size reduction. Error is reported by the order size stays the same.

Also:

Got same error today, also on FA. In my case I didn't get any shares on this order at all (all other orders executed fine).

Also wonder what is means and how to avoid it.

One thing I noticed, that at that point of time I could had been pretty close to margin. Wonder if IB decided that I don't have enough and tried to "reduce" the order accordingly?  

[A]  by Alex btccplayer@gmail.com View/Reply Online (#45748) 

Added on 01-Nov-2020

I got reply from IB:

"We confirm the "errorCode: 481 errorMessage: Order size was reduced" call back is due to IB not having any

shares for the customer to borrow for short selling, but you don't need to worry about that your orders didn't

get rejected as we had allocated the shares you intended otherwise your orders would get rejected by server."

So probably nothing to do

[Q] (undocumented, seen on may 2005) Error 505

[A] https://groups.io/g/twsapi/topic/4040516#4368

[Q] Errorcode(???) Order Canceled – reason:Order size exceeds amount allowed by fat-finger check

by eugene.kononov@gmail.com

View/Reply Online

[A] (29-Mar-2018)  this error can't be overridden through the precautionary settings in TWS/IB Gateway. There is a precautionary setting for order size, but it is separate. If the user contacts IB ahead of time they can request to have the fat finger check modified in their (live) account.

[A] Incorrect answer by rwk was

I'm surprised you haven't had this error before.  On the TWS click:

Edit > Global Configuration > Presets > Futures > Precautionary Settings > Size Limit > [set to zero (default is 5)]

[A2] by Frank Bell

I had problems with this in a paper account today.  I had a short position of 75000 TNA.  I had orders to buy 30000 at the market canceled even after changing the presets.  Sometimes it would accept orders at a given size and then later reject.  I think this is just another example of paper account flakiness.

[Q]  (undocumented, seen on Sep 2017, Apr 2020) Error 10187

[A] https://groups.io/g/twsapi/topic/6076358#44164

[Q] Errorcode(10197): No market data during competing live session

Last night TWS updated itself.  When I started my program to collect data from my paper account I got an error "No market data during competing live session" for just a forex request.  However everything was working ok.  Overnight I got the same error for all requests and they had all stopped working.  

From the api.log file

04:26:00:868 -> —:4-2-16-10197-No market data during competing live session

I should point out I logged into my live account this morning on the same computer and everything looks fine.

Has anyone ever had this error?

[A] by Josh, View/Reply Online (#41512)

Added on 15-Jan-2021

The message is better thought of as an indication of a delay in establishing connection, or a disconnection from the market data farm, rather than a cause of disconnection. The time when data is eventually returned depends on the underlying reason for the disruption. A disconnection would certainly happen during the nightly server restart (around midnight est in the US), which can vary in duration depending on amount of maintenance necessary. After the restart generally (but not always) reqMktData subscriptions are expected to eventually continue again automatically. Market data for different instrument types is hosted on different servers around the globe which have different reset times. So I would suggest looking to diagnose the reason for disconnection, or delay in connecting, to a particular market data farm and that will indicate why the message is being received.

Also, not really an answer, but that error mentioned (and ignored? hm…) here: View/Reply Online (#45987)

Your message contains an 8 step overview. I would like to emphasize that step #5 must contain two activities: 5a. cancel market data subscription for the tickerID: cancelMktData(1) 5b. disconnect socket connection: eDisconnect() Unsubscribing from all that you were subscribed to before closing the connection is necessary to ensure that IB's servers understand your intentions, and that the used tickerID values are released and can be re-used.  Having said that, I sometimes also get a error "10197   No market data during competing live session". However, I don't use a paper trading account. Instead, I receive this in my live trading account, even though there is only one account active. Thus far this error message has not affected my automated trading, so I simply ignore it.

Thank you, J G. Calling cancelMktData(<ticker_id>) before calling disconnect() fixed the issue, both with live and paper trading.

I'll keep monitoring for error 10197 on the live trading account to check if it is still returned occasionally as you mentioned.

[Q] Errorcode(10225): Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately

[A] found here https://groups.io/g/twsapi/topic/69943076 and https://170.106.50.235/mementum

Up to the TWS API v972, IB realtimeBar subscriptions could stop working during the trading session and no error notification was sent to the application. This happens because of what's called 'busts' events.

Quoting the response in "TWS API Users Group" question:

"The busts occur when the datastream is interrupted or modified in the middle of a bar and the subscription is cancelled by the backend. They should be relatively uncommon but unfortunately no can't be avoided completely"

In TWS API v978,a special error code was added to notify the application about the bust event so that the subscription could be renewed. Below is the example of such error message (from TWS Gateway API logs):

15:24:00:896 -> 4-2-16777221-10225-Bust event occurred, current subscription is deactivated. Please resubscribe real-time bars immediately.-

Update: after a week of testing – realtimeBars seems to be working flawlessly. Re-subscribing after bust events doesn't require any delay to be added by the application – just make sure to cancel the previous subscription before re-subscribing, as was mentioned above.

[Q] Errorcode(1101, 1102): Error validating request:-'rc' : cause – You must specify an account.

[A]  by Ed Gonen ed.gonen@gmail.com   (added on 30-Jul-2015)

From my experience the midnight TWS disconnects happen at the first 30 minutes after the midnight. I think very close to the 20th minute. Your 5:21 London exactly matches that situation. My logs usually show the 14th minute – i.e. 12:14.

I don't think you need to disconnect from TWS in this situation. At least – I don't. The code indicates a disconnect between TWS and IB servers and TWS always automatically takes care on that. This lasts about 1 min. and you get 1102 code. This is based on many years experience with IB.

——————— also see  API Message Codes  —————————-

System Message Codes

Code

Description

1100

Connectivity between IB and TWS has been lost.

1101

Connectivity between IB and TWS has been restored- data lost.*

Reminder: when you get 1101, you have to re-establish all your market data connections

1102

Connectivity between IB and TWS has been restored- data maintained.

1300

TWS socket port has been reset and this connection is being dropped.

Please reconnect on the new port – <port_num>

*Market and account data subscription requests must be resubmitted i.e. when you get 1101, you have to re-establish all your market data connections


[Q] Error: 2109 Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed.

[A] by Josh View/Reply Online (#40645)

It is actually just a warning to let you know that for the designated order type and exchange there is no distinction between rth and 'outside rth'. It will not prevent an order from being placed. For stocks, this is because regular stop orders don't trigger outside of the regular trading hours period 9:30-4:00. This is to protect traders, as the extended trading hours can be very illiquid and the usual NBBO rules don't apply. You can however place a stop limit order which would be active outside of rth (though there's a reasonable chance of a very bad fill with this combination).

[Q] error 2137 The closing order quantity is greater than your current position

[A]  by Scott Hodson shodson at gmail com

Added on 02-Aug-2018

View/Reply Online (#40990) 

For future reference message codes documented here:

https://interactivebrokers.github.io/tws-api/message_codes.html

2173 = Cross Side Warning: This warning message occurs in TWS version 955 and higher. It occurs when an order will change the position in an account from long to short or from short to long. To bypass the warning, a new feature has been added to IB Gateway 956 (or higher) and TWS 957 (or higher) so that once can go to Global Configuration > Messages and disable the "Cross Side Warning".

[Q] Order size is too large IB limit is 35000

boy, I hope I’ll face it some day… :))

[A] by Frank Bell (from this thread)

Size limits vary based on exchange, legal, and IB internal limits. If you are

bumping up against an IB limit they might change it for you. Contact IB.

[Q] First order after IB GW API connection causes disconnect and order is ignored

Added on 27-Apr-2015

[A] this thread

[Q] Historical Market Data Service error message:No historical market data for XAUUSD/CMDTY@CMDTYBBO Last 60

[A] View/Reply Online (#44172)

Added on 03-Apr-2020

Commodities don't have trades data ('Last' in the error message) so you have to choose bid, ask, or midpoint as the data type.

If you check in the TWS chart you'll see that a different data type is selected and 'Trades' is not one of the options.

[Q] While trying to modify submitted order getting: "Unable to modify this order as it is still being processed."

Added on 18-Jun-2020

Hello,

I've run into an issue when trying to modify an order's limit price in quick succession after initial entry.  The error is:

"Unable to modify this order as it is still being processed."

While I do keep track of the latest limit price submitted on my end, I have no way of directly ascertaining what the last accepted value was in TWS.  (Other than receiving the error message and deleting my last recorded limit price.  This seems like a terribly convoluted approach and ripe for errors.)

So, there are two ways I can think of addressing the issue below (in order of preference):

1. Get the last known m_lmtPrice from the API.  openOrders() provides it but that doesn't fire after each newly submitted order.  orderStatus() does fire after each new order but doesn't appear to include a way to get m_lmtPrice.  Is there another way to get it?

2. I can forget tracking the limit price directly and simply wait until it's "safe" to send a modification based on the order status.  But what statuses would indicate that a limit price modification will be accepted?  PreSubmitted? Submitted?  Inactive?  What if I want to modify the limit price a second time…will the same logic work?

Appreciate any insight on the above.  Part of me is hoping that I'm being a knucklehead and missing a simple solution staring me right in the face.

[A] View/Reply Online (#44735)

In case anyone runs into the same issue in the future, I'm going to post what I found here for posterity's sake.  Essentially, I couldn't find a way to reliably get the order limit prices without constantly calling reqOpenOrders().  That was too inelegant for the way I'm set up, so I went with #2 above and waited for the order status to go "Submitted" before attempting modifications.  That alone was sufficient for my purposes, so I didn't tinker with "PreSubmitted" or "Inactive".

—— Also ———-

In case it helps, if you are using stop limit orders it is normal for the status to stay at PreSubmitted until the order is triggered (you will see a 'whyHeld' attribute with value 'trigger'). In this state it is possible to safely modify the limit price, just FYI.


ContractDetails

[Q] What is the difference between contractDetails.liquidHours vs contractDetails.tradingHours?

[A] View/Reply Online (#45684)

Added on 04-Jun-2021

The contractDetails.tradingHours defines the times when the instrument is open for trading. These times are set by the exchange(s) where the instrument is being traded.

contractDetails.liquidHours is defined by IB. For a number of instruments (not all!) have they investigated when trading in this instrument is liquid. I do not know what criteria IB uses to define this though. Outside these liquid hours are your trading options limited by IB. For example: MKT orders are not accepted as the spread may be large during these non-liquid hours.

[Q] Exchange list in ContractDetails?

I'm trying get exchange list for future symbols using this code:

Contract contract;

contract.symbol = "ES";

contract.secType = "FUT";

client->reqContractDetails(reqId++, contract);

Then in the callback contractDetails(), when I printed contractDetails.validExchanges, it shows: "GLOBEX,MIBSX".

Whtat is this "MIBSX"?  Also why is "CME" not listed (if you try to add ES in TWS, it tells you both GLOBEX and CME)?

thanks,

–Zhao

[A] by rholowczak (from this thread)

GLOBEX is CME's electronic market: http://www.cmegroup.com/globex/

[Q] Change of Timezone specs in contract details?

What's up with timezones in TWS API contract details today? I've got all my algos messed up today on startup due to what is obviously a change of Timezone specifications. I use a custom dictionary to convert from proprietary IB timezone specifications to IANA timezones to get trading hours for my task scheduler and work within a common standard. US stocks in the API are now in a timezone named "US/Eastern". Didn't see it coming and can't figure out if it's a welcome change to some standard specification (not IANA though since that would be "America/New_York"), or to yet another proprietary format that needs to be guessed?

[[A] View/Reply Online (#45898)

Added on 03-Dec-2020

It seems that IB recently changed their format.

Too bad thay didn't seize the opportunity to go with IANA standard, but at least they are documented here: they look like a mixed bag of IANA, legacy and three-letter codes.

— also

The time zone IDs seem to be from the tz database: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

From python pytz can handle them: https://pypi.org/project/pytz/

OP: Yes indeed the IDs are in the database though roughly about half of them are deprecated. Unless these get completely removed from the db this still looks like a good step towards elimination of the poorly undocumented IDs used in the API previously.

[Q] Incorrect Mintick?

When I run reqContractDetails for the stock symbol DGLY, I get a mintick back of 0.0001. However, in active trading I am only able to trade with 0.01. This throws off my code as I am trusting what TWS is telling me with the mintick. Any idea on what could be going wrong?

[A] by Richard L King

Added on 27-Feb-2021

Each contract details contains a list of market rule ids, one for each exchange in the 'valid exchanges' list. A market rule tells you exactly how the minimum tick varies by price. You get the details of the market rule from the reqMarketRule() function.

 

For example, for DGLY the market rule for all exchanges is the same and has id 557. Calling reqMarketRule(557) in my API test program returns the following:

 

20200912 14:53:36.070   ==== Market Rule Begin (marketRuleId=557) ====

LowEdge=0, Increment=0.0001

LowEdge=1, Increment=0.01

==== Market Rule End (marketRuleId=557) ====

 

This shows that for price >= 0 and < 1, the min tick increment is 0.0001. For prices >=1 it is 0.01.

 

A very handy mechanism!

 

Richard

The relevant rules for exchanges SMART,TSEJ,CHIXJ and JPNNEXT are 837, 789, 837 and 942. This is what I get back for them:

 

==== Market Rule Begin (marketRuleId=837) ====

LowEdge=0, Increment=0.1

LowEdge=5000, Increment=1

LowEdge=100000, Increment=10

==== Market Rule End (marketRuleId=837) ====

 

==== Market Rule Begin (marketRuleId=789) ====

LowEdge=0, Increment=1

LowEdge=3000, Increment=5

LowEdge=5000, Increment=10

LowEdge=30000, Increment=50

LowEdge=50000, Increment=100

LowEdge=300000, Increment=500

LowEdge=500000, Increment=1000

LowEdge=3000000, Increment=5000

LowEdge=5000000, Increment=10000

LowEdge=30000000, Increment=50000

LowEdge=50000000, Increment=100000

==== Market Rule End (marketRuleId=789) ====

 

==== Market Rule Begin (marketRuleId=942) ====

LowEdge=0, Increment=0.1

LowEdge=3000, Increment=0.5

LowEdge=5000, Increment=1

LowEdge=30000, Increment=5

LowEdge=50000, Increment=10

LowEdge=300000, Increment=50

LowEdge=500000, Increment=100

==== Market Rule End (marketRuleId=942) ====

 

The closing price was 755, so the minticks are

 

SMART                     0.1

TSEJ                           1

CHIXJ                             0.1

JPNNEXT                  0.1

 

It wouldn't surprise me if some of the rules have errors, but I've found this mechanism to be pretty reliable, and it's the only way to make any kind of sense of stocks on the LSE, which have all sorts of different rules.


Indexes

[Q] Is there a way to get a list of all indices available on IB? (aka "Product Listings")

[A] by paul_blinn@hotmail.com View/Reply Online (#49151)

Added on 19-Apr-2022

You might try searching by region and exchange:

Product Listings – Indices – North America | Interactive Brokers U.K. Limited

Placing Orders (common)

[Q] What order types are supported by IB?

[A] See official documentation on all the available order types:

http://interactivebrokers.github.io/tws-api/orders.html

[Q] I sent an order, what should I expect next?

[A] by skunktrader2001

After you sent your order TWS will ASYNCHRONOUSLY send you one or more orderStatus() and openOrder() callback messages as the order is processed (PendingSubmit -> PreSubmit -> Submitted -> Filled), as well as execDetails() callbacks as the orders fill. You will need to capture these state messages from the callbacks for each order in some type of data structure, which you can then check on each iteration through the loop to see if there have been any fills / cancels etc.

————— Also by Richard L. King ———–

Regardless of what TWS logs, your own app should log every event concerning orders: thus, log placeOrder, cancelOrder, and the openOrder(), orderStatus(), and execDetails() callbacks. And you absolutely must log errMsg() events. You should be able to determine from your log exactly what the current state of an order is.

[Q]  How can I trade stocks during the pre-market or post market?

[A] on IB website:  How can I trade stocks during the pre-market or post market?

Note: most importantly: you can't have a market order or anything that turns into a market order be a part of an outside RTH order.

(quote from the IB website linked above)

* For stocks, Market orders and Stop orders (which, when triggered, become Market orders) are only active from 09:30 until 16:00 EST (the Nyse's regular trading hours session). To have the order active in all sessions including the Premarket hours, Regular Trading Hours and the Aftermarket hours, you must use a Limit or Stop Limit type order and add the attribute "Allow Outside RTH". The same settings apply for Globex/ECBOT/Nymex Future contracts. The Nybot accepts Stop orders and converts them internally to "Stop with Protection" orders that are active for all available hours. A Future contract's RTH session is detailed in the contract's description window.

* Forex, OTCBB, and Pink Sheet securities do not have a Regular Trading Hours designation as there is no centralized marketplace or primary exchange.  Traders should use caution when trading these products during the early and late sessions as orders will execute regardless of any default RTH restriction previously applied.

* Outside Regular Trading Hours for IBKR Pro accounts begins at 4:00 EST and lasts until market open, and restarts at market close until 20:00 EST. Outside Regular Trading Hours for IBKR Lite accounts begins at 7:00 EST and lasts until market open, and restarts at market close until 20:00 EST. Regular Trading Hours for both account types are 9:30-16:00 EST.

[Q] execDetails vs orderStatus

[A] by man910 View/Reply Online (#46335)

Added on 06-Jan-2021

You should monitor execDetails instead of orderStatus.  I had some errors with a couple of my orders and thought that orderStatus would give me better information, but it often times didn't get called.  I ended up having to look for specific codes in the error callback for those order failures.

Important notes concerning IBApi.EWrapper.orderStatus :

  • Typically there are duplicate orderStatus messages with the same information that will be received by a client. This corresponds to messages sent back from TWS, the IB server, or the exchange.
  • There are not guaranteed to be orderStatus callbacks for every change in order status. For example with market orders when the order is accepted and executes immediately, there commonly will not be any corresponding orderStatus callbacks. For that reason it is recommended to monitor the IBApi.EWrapper.execDetails function in addition to IBApi.EWrapper.orderStatus.
  • Beginning in API v973.04, a parameter mktCapPrice is included in the orderStatus callback. If an order has been price-capped, mktCapPrice will indicate the price at which it has been capped.

[Q] How do I query the status of my orders?

[A] by Dmitry

Basically you don't need to query the status of your orders. IB will notify you. See the IB's API docs: orderStatus()

This f-n is called by API for you (not by you:) whenever the status of your order changes and/or fired after you reconnect to TWS.

Also see the following example, which demonstrates how the orderStatus() event behaves when there is a partial fill of an order. (update (Feb-2021): example link is broken, had to dig "following example" content from google cached pages):

(quote)

Partially executed order status

The following example illustrates how orderStatus() (order status) will behave when an order is partially executed.

Partial execution example

You place an order for 1000 shares of XYZ stock. There were four different executions before the 1000 shares were all executed. The first part is executed for 200 shares, and then the second part is executed for 200 shares. The third part will be executed for another 200 shares, and the final part will be executed for 400 shares.

  • First execution: 200
  • Second execution: 200
  • Third execution: 200
  • Fourth execution: 400

In the orderStatus() activity, according to this example, the API will continuously receive status information as:

carried out

status information

First execution

Status = number of submitted executions = 200 remaining number = 800

Second execution

Status = number of submitted executions = 400 remaining number = 600

The third execution

Status = number of submitted executions = 600 remaining number = 400

Fourth execution

Status = number of submitted execution = 1000 remaining number = 0

(end of quote)

Also see execDetails(), which you can request by reqExecutions()

——-

Also: Answer by Ewald (View/Reply Online (#189) from https://groups.io/g/insync/

What you are looking for is the openOrders (or openTrades which has more information) and positions methods. These methods are much faster and also more reliable since reqOpenOrders can report stale information. In general it's best to not use reqOpenOrders, reqPositions or reqExecutions; instead use the non-request variant.

[Q] Retrieving past trades in an account

[A] by ds-avatar View/Reply Online (#49087) 

Added on 22-Feb-2022

In TWS API you can only retrieve trades for the current trading day.

You can use two approaches:

The first one is to request executions, i.e. reqExecutions. Pros: it is well known and seems to be widely used, and can consume executions filters (including contracts traded). Cons: even with a filter it could still be pretty verbose because it can return numerous partial fills per order which the client needs to accumulate and collate.

The second one is to get completed orders, i.e. reqCompletedOrders. Pros: it returns concise trade data per each order. Cons: despite being in API releases for at least 2 years it's still poorly documented, and it spews all completed orders after each request so any filtering must be carried out by the client.

For trades going back beyond 1 day, reports should be downloaded from the account management website (or set up for automatic delivery by email).

– also by Jürgen Reinold:

In a similar topic last year, Richard King pointed out a tweak in TWS that allows you to retrieve execution reports for up to seven days. This does not work for IB Gateway, though. In TWS go to File -> New Window -> Trade Log and change the pull-down "Show trades for" from "Today" to a longer period. The maximum is "Last 7 Days".

[Q] How to distinguish Client ids from API vs non-API?

[A] by Josh View/Reply Online (#41871)

Added on 04-Mar-2019

Orders placed by other routes such as IBKR mobile or manually from TWS will always have client ID = 0. Unlike orders placed from the API, they will also have an API orderID of 0 unless they are bound from the API. If you want to distinguish your API orders you could set the client ID to a different value when you connect (unless your API client needs to modify/cancel non-API orders) or use the orderRef field in the Order object to label them with a String.

Update (Feb 2021): orderRef only survives "session", also you can use permID (survives forever). See this question for more details: [Q] Order field to identify a strategy? (clientID vs orderRef vs permID)

[Q] Where to Set GTC in placeOrder API?

[A] by rwk2095 (src)

It's part of the IOrder object:

timeInForce() As String

The time in force. Valid values are: DAY, GTC, IOC, GTD.

http://www.interactivebrokers.com/php/apiUsersGuide/apiguide.htm

[Q] "update order" VS "cancel and creating order"?

[A] view online

Added on 24-Feb-2021

Better to edit:

1.Faster (one operation instead of two)

2. you keep your orderid

3. you also keep your OrderRef, if ever you use it to track/classify your orders

4. If ever you have linked orders (OCA), you keep them linked. If you cancel the parent order, all dependent orders will be cancelled too, and you need to re-create them all

Also note:

Modifying or cancelling "directed" API orders incur fees. SMART routed orders do not. See next question for details: [Q] How often can I modify an order?

[Q] How often can I modify an order?

[A] by JR

Added on 24-Feb-2021

Modifying or cancelling "directed" API orders incur fees. SMART routed orders do not.

But executions get you credits so you only actually have to pay fees if your ratio of (#Orders Cancelled or Modified / #Orders Executed) > 25

From https://www.interactivebrokers.com/en/index.php?f=14718

Cancel/Modify Orders

United StatesUnited States flag

Asset/Order Origin

Routing Venue

Fee per Order 1

Execution Credit 2

API/CTCI Smart

N/A

None

N/A

TWS Directed and Smart

All

None

N/A

API/CTCI Directed

All

USD 0.01

USD 0.25 per API/CTCI Directed execution

Orders with a time in force that specifies an expiration time, such as GTC, will be assessed cancel fees according to the above schedule.

Click here for cancel/modify fee examples.

Europe Middle East Africa (EMEA)European Union flag

Asset/Order Origin

Routing Venue

Fee per Order 1

Execution Credit 2

Direct Routed Order

TGATE

EUR 0.50

EUR 2.50 per TRADEGATE execution in the same product on the same day

Notes:

  1. Unless otherwise specified, the per-order fee applies to both order cancellations and modifications.
  2. Execution credits are applied against cancel or modify fees for the day. Execution credits cannot exceed cancel/modify fees.

[A2] by Rob Terpilowski rterp99 at yahoo.com

Added on 18-Feb-2016

IB will freeze your account if you send too many order modifications relative to the number of actual executions you are getting.  I had an MM strategy that was doing about 15+ order modifications for every 1 execution it received.  IB sent me a notice within a few hours of it running saying that if I didn't disable the strategy they would be disabling all orders originating from my account.  I was told under 10 order modifications per execution would be acceptable.

[Q] This account cannot have open orders(including Combo) on both sides of the same US Option contract in related accounts.

[A] by Josh View/Reply Online (#43818)

Added on 15-Feb-2020

It sounds like you're interested in having orders held on the IB server as 'inactive' and only validated by the server for all possible errors at a later time, not at the time of submission. Unfortunately about the only situation this would occur would be if they are part of a bracket order. In other cases orders will be checked immediately and rejected if there is a problem such as existing orders on the opposite side of the same option contract, even if there is some condition attached to the order preventing it from being submitted immediately to the exchange.

If the orders are complicated and can't be part of a bracket order the only possibility is to have them held on your local machine, as you describe. If you're concerned about the stability of your connection you could possibly use a dedicated virtual server, such as AWS. Or you could use similar options on the same underlying for hedging if that could work with your strategy.

[Q] How to change the limit price of an order without changing order size after partial fills?

… to change the price of the order, I would place an order with the same order ID, with a different price, but keeping everything else the same.  But in case of partial fills, what happens if the quantity is the same with the changed order?…

…..Basically – do I need to adjust the quantity of subsequent changes to the order depending on how much quantity has already been filled?  Or is the quantity always the total of this entire order?…

[A] by Josh jb201448 found here

Added on 26-Apr-2016

Hi Yu,

Yes the quantity of an order is the total quantity, including those contracts which have already filled.

So if 50 of 100 shares fill and you want to just change the limit price of the remaining 50, but not the number of shares remaining, you would just adjust the lmtPrice and the totalQuantity value would remain 100.

-Josh

[Q] I’m getting multiple filled orderstatus updates

[A1] by vanx23@outlook.com

The "Filled" order status may be triggered multiple times for the same order.

One example of that is when IB's routing logic decide to split your original order into smaller amount that would executes in a short burst. The end result is you will have your order filled as a whole, but will have an "Filled" order status for each sub-parts that are been filled, with the filled amount showing as the original order size.

If the filled size match the original order size, then yes, you can use a flag to avoid processing the redundant "Filled" order state.

[A1] by Ed news1000@edshome.net

Also note that there can be duplicate order status events – you have to detect these yourself.

It might be easier to use the Execution callback instead. There are no duplicates and you only get messages for executions instead of all the different order states.

[Q] Ok! Order executed! But what was the fill price?

[A] by Vlad Palnik (it was offtopic on this thread)

see execDetails().execution.m_avgPrice

[Q] Order Id mismatch between API and TWS

Can someone please explain why the Order Id displayed in TWS order's attribute is different then the order Id which is returned from PlaceOrder command in API? How can it be matched?

[A] by Josh View/Reply Online (#42922)

Added on 14-Sep-2019


Unfortunately the API Order ID is not available within the TWS display, only in the audit trail. The 'permId' is the order ID which is assigned by TWS after an order is placed, and will be unique across the account.

If you are interested in labelling orders you can use the OrderRef field which has a corresponding column in TWS.

[Q] Why my order has “Inactive Order” Status?

[A] by Kurt (src thread)

First, if you requested to *transmit* an order but it comes back as

Inactive, you were probably ignoring the results from the error callback.

Conceivably this could happen without an associated error (you never know

with IB) but I haven't seen it.

IB's treatment of order states and reporting is only barely documented, has

changed over time, arbitrarily without notice and with no change to

documentation.

That said, Inactive may mean more than one thing.

(1) Inactive used to always mean basically one thing: the order is not

active yet. This used to include orders "placed" with transmit false, for

which orderStatus messages used to result. Such orders no longer produce

orderStatus messages. When IB changed that it broke my code. As a result I

had to create my own order state to represent orders placed with transmit

false. It is not ideal because I like the acknowledgement that the order

was received (by TWS) with certain fields pre-checked and more likely ready

to go. The current behavior weakens some aspects of reliability.

(2) Inactive also used to be used for orders that were transmitted but

rejected (e.g. due to some bad field) but reusable, i.e. you could correct

and retry the transmit. Useful for interactive apps or automated apps that

provide interactive fallback. I believe some order rejections still work

this way but I'm not 100% sure.

(3) Inactive can now mean something else: the order was rejected and the

system (for no good reason) is not letting you correct and re-transmit it.

The order id is dead and can not be reused. IB introduced this new behavior

some time in the last year.

In at least some cases you can distinguish (2) from (3) based on the error

code.

My code currently uses an artificial order status string "Rejected" for case

(3), and sets that order status if error code 201 comes in. Error 201

produces the message:

"Order rejected – reason: …"

Status "Rejected" in my interactive app basically tells the user not to

bother correcting the order and re-transmitting. (I should permit correct

and automatically generate a new order id.)

You might prefer for simplicity to just waste the orderId and create another

one. That is probably more likely to continue working if IB continues to

mess with things.

In any case if an order is Inactive you do not need to cancel it.

To understand more in a particular scenario you can just try things that

produce errors, and look at if/how the order appears in TWS as well as any

code from the error callback.

-Kurt

[Q] How to cancel all orders?

[A] by rholowczak (form this thread) (added on 16-Jul-2015)

The "nuclear option" is reqGlobalCancel()

[Q] How to cancel all open orders and close all positions?

[A] by Josh View/Reply Online (#40698)  (added on 12-Sep-2015)

Please note there is actually not a single function to 'close all positions' from the API. It would be necessary to receive the current list of positions from the position() or updatePortfolio() callbacks, and then create and place closing orders. There is a reqGlobalCancel function to cancel all orders.

[Q] How to pause order (like in TWS)?

[A] by Josh View/Reply Online (#39988)

Added on 06-Jun-2018

Unfortunately there's no way an order can be paused from the API. It needs to be cancelled and re-placed. This is essentially what TWS does behind the scenes anyway.

Josh

[Q] How to get order history?

[A] by Josh View/Reply Online (#43808)

The TWS (socket) API provides trading history for up to a week, IB gateway is for the current day only; if you need history further back than that you could use flex queries (which don't require username/password) or the CP API.

[Q] How to track ALL orders (including cancelled orders, and including manually created via TWS when my program was not running)?

Hello,

I am trading both manually and via the TWS API.

In order to analyse my trades better, I am looking for a way to

list orders that I cancelled either manually or via the API.

My problem is that the API-connected program is not running

all the time, so when I do the occasional manual cancel of a STP-order

it is not recorded by the API-connected program.

I could not find a way to export cancelled order Information from TWS.

Ideally I would like to export a list of orders like the one via Account – Trade Log

Menu entry in TWS.

Does anyone have an idea of how I could get a consolidated list of

all cancelled orders within a timeframe, meaning both manually cancelled

and automatically cancelled orders?

As I said, just protcolling them via the API-connected program does not

work, since it is only started sporadically throughout the day. Sometimes not

at all (e.g. days, when I only trade manually).

Thank you very much for any help in advance.

Best,

Josef

[A] by Josh View/Reply Online (#39971)

Added on 06-Jun-2018

Josef,

To receive order history I would also suggest taking a look at the audit trail, it should be easier to parse than the log file and will show order cancellations.

https://www.interactivebrokers.com/en/software/tws/usersguidebook/realtimeactivitymonitoring/audit_trails.htm

-Josh

——- update from Josef: ————

I know this is getting off-topic regarding the API, but I thought this

might be helpful for someone with a similar problem:

The audit data, as Josh has generously indicated, can also be

found in separate xml-files named

Mon.audit.xml etc. for each weekday.

They can be parsed easily and the benefit is that this can be done

by a script by just accessing the files. So one does not have to open

the audit menu in TWS which can be less automized (though I am sure

the marvelous creators of IBController could even automate this).

All the audit menu item does is create an html file from the data in the

xml-files which is then auto-displayed in the standard browser.

The files can be found in the infamous folders under the jts-folder

that specify accounts by a series of letters.

They are overwritten on a rolling basis, so saving them via a cron-job

makes sense.

Thanks again,

Josef

[Q] I got multiple execution messages.

>>> I tried this morning and still got multiple execution messages for

>>> single executions – sometimes. Sometimes it was one execution message

>>> for one execution. But I did get fewer execution messages than order

>>> statuses (stati?)

>>>

>>> Anyone else get multiple execution messages? If they get doubled, they

>>> seem to be 3-5 ms apart.

[A] by Kurt (form this thread)

>> I could be missing something re the "multiple" messages, but will hazard a

>> guess.

>>

>> You should never get two execution messages with the same execId. You have

>> to keep track of such things if you want any reliability. Duplicate

>> messages should not be a problem whether they are orders or executions as

>> long as you maintain the necessary information. (Well I can't speak for

>> what is possible within Excel.)

>>

>> Some people think it is trouble to keep track of a state for each object

>> they are concerned with. But the trouble of not doing so is immense. You

>> create problems that don't really exist. IB is not trying to create the

>> kind of system that would be reliable to use without keeping track of

>> state

>> at the other end. So if you don't maintain the necessary information, the

>> problems you get as a result are essentially problems you created.

>>

>> You can request executions at any time, and then you will get more

>> "duplicates", but again just ignore the ones you already received. As I

>> recall some people request executions explicitly because they are seeing

>> execution messages getting dropped. (This is a reason to track both orders

>> and executions since conceivably either might get dropped.)

>>

>> -Kurt

> I wasn't logging execID; I'll start looking at that. There doesn't

> seem to be a corresponding unique identifier for order status – is there

> one?

>

> I do monitor all state on my end and keep my own copy of the info. I

> imagine it would be crazy-making to try it any other way. I don't

> consider a trade closed until I get both the order status update of

> 'filled' *and* the portfolio update that shows the correct position.

>

> I have never sent a call to get executions – they just show up after

> each order.

The order id fields tell you what you need to uniquely identify the order

you placed and associate the resulting activity with that order. Both

orderStatus and openOrder messages are involved, and there is a little

redundancy between the two, as I recall. (As I recall, they always come in

pairs, and always in a certain order.)

Sounds like you have a generally good approach. I was going to mention

portfolio updates, but didn't want to get into it. 😉

You may have to request executions or open orders if you have to quit and

restart your app, depending on what state you save to disk. Or you may want

to do it if you find you are missing executions or orderStatus messages, as

many people here have reported happening.

-Kurt

[Q] How to submit a sell order as soon as a buy order is executed

Hi all,

I would like to submit a SELL order as soon as my BUY order has been executed…is there a way to submit this logic in one shot or do I have to listen to the execution of the BUY order in order to submit the SELL order?

any help would be really appreciated…

Stef

[A] by tdrtw from this thread

I haven't looked into this myself yet, but I believe there is bracket orders which may suit what you want to do.

I presume you know what price your entry will be and know your initial stop point BEFORE it happens. You can tie a bracket order which will be your stop to your entry order. The bracket order will not come into effect until your entry order is fired.

Let us know what you find out.

Cheers,

Damien

[A2] by Ed

You want parent/child orders. The parent is your entry and the child

is the exit. In the child order you set ParentId to the entry order.

There may be some discussions in the archive. I have used this and it

actually works (not always the case with the api).

[Q] How do I know if my stop order is triggered from API message?

[A] by Richard L King (found in this thread)

You get orderStatus and openOrder callbacks when a stop limit order is triggered.

You should read the definitions of the various statuses in the API documentation to find out which status applies (I think it changes from presubmitted to submitted but you’ll need to verify that).

When it comes to orders, you should be logging absolutely everything that happens – you can’t afford to miss anything where your money is at stake. If you were doing this, you wouldn’t have needed to ask the question.

Regarding how to get the best execution price with a stop limit order, I doubt anyone knows any better than you, and it probably depends on whether you’re trying to get into a position or out of one.

Richard

————————————— also ——————–

Joe: Richard,

You are correct. The orderStatus change the 'PreSubmitted' to 'Submitted' once the stop order is triggered. This something I missed. Thank you for the information.

For the limit order execution issue, I set up an stop limit order for cut loss instead of using stop order which usually results in a very poor price. So, do you think it is a practical approach to create a thread which keep comparing the last price with my limit order price if the last price cross my limit price, we re-submit the limit order with new limit price which could be possible executed. I am pretty new to this, your help is best appreciated.

Best,

Joe

————————————– and finally ———————-

Your suggestion is one possible way, though in a fast-moving market you

might find that you are chasing price movement and never quite catching it,

and you end up incurring a much bigger loss than if you had used a stop

order. Bear in mind that the prices you're getting through the API may

easily be 100+ milliseconds behind the market, and it could easily be

another 100+ milliseconds before your amended order reaches the exchange.

Another possibility is to set the limit price of the stop-limit to be the

worst price you're prepared to accept, and the chances are you'll get a

better fill than that if the market is currently higher (for a sell) than

your limit (in which case a market order would have done just as well). But

again, you may not get filled at all.

In a stop-loss situation the important thing is to be out of the position.

If the exchange offers a native stop order, my view is that it's the best

option – it will get the best price that's available at the time. There may

be a better price available a microsecond later, but there's no way you

predict that.

Where the exchange doesn't offer a native stop order, IB usually has a

simulated stop order which automatically does rapid order amendments with

stop-limit orders in the way you described – but much more effectively than

you can do it yourself, because IB are very close to the exchange.

If you're trying to get out of a profitable position, rather than cutting

your losses, there is perhaps more room for creativity. You could have a

limit or limit-if-touched order to extract the best profit, and a stop or

stop-limit order as a backstop: you could then adjust both of them

periodically, perhaps converging them together – but not too close, or you

might find they both execute (even if they are in an OCA group).

The possibilities are endless, and I pretty much guarantee that there are

others on this forum who will have a different view from mine. There's no

'right' answer.

[Q] Should I use IB’s trailing stop and other advanced order types?

Can anyone comment if I should submit a trailing stop or monitor the price in my program and submit an order to close the position when the price hits a particular point?

[A1] by Jason Sapp

My personal preference is to always have a stop order in the market (or at least simulated at IB's back-office), versus just exiting when price hits a certain point. If my ATS goes down, then I know I'm still protected. Lastly, some of the trigger methods that IB supports take a little time to program correctly and they've already done it for you, so that's worth something as well.

Now, with all of that said, I have implemented my own trail, but I still use regular IB stop orders. It's just that I change the price when my algorithms dictate.

Jason

[A2] by Ed <prize@…>

I'll second that idea.

I use the parent/child facility to specify my entry order, profit exit

and stop exit all at the same time. Either I will have no position or I

will have a position with a profit and stop. Should my ATS be 'down'

(anything from program crash to power failure or isp failure) I will not

be exposed to any disastrous drawdown.

As has been mentioned, you don't have to post your orders at the price

you ultimately intend. You can adjust things as you go along (although

not every aspect of a placed order can be changed).

For me it winds up being the both of best worlds, at the cost of a

little more programming. As always, only you can decide which approach

is best for your particular needs.

[Q] How to let an order to execute at a particular time ?

[A]

jimshaw4585: One can combine "good after time" and "good till time" to create a narrow window.

tdrtw: I use a "Sell on Market, Good After Time set to 1 minute before market close" to liquidate any open position grouped in a One-Cancels-All with my other exit orders. In the last 4 years it has worked as anticipated.

[Q] OCA orders?

[A] by Kurt (from this hread)

Orders don't cancel each other unless you put them in an OCA group. That's

what an OCA ("One Cancels All") group is for.

I don't see you setting the ocaGroup field anywhere.

Also don't forget to set ocaType according to what you want.

-Kurt

——————— but then also this: —————————-

When you create a bracket order (ie an entry order together with a related

stop order and/or a target order), you don't have to set the OCA group,

because TWS does it for you (ie orders that have parentId set are

automatically placed into an OCA group). Indeed if you do set the OCA group

TWS overwrites it with its own value.

Bruce, it would be a lot clearer (though not strictly necessary) if you

created a new order object for each order. For the stop and target orders

you need to set parentId to the orderId for the entry order.

Richard L King

todo: give couple working examples of OCA done manually (by Kurt) and by Richard

using a parent/child arraignment

automatically creates an oca group.

————————————— and then very important from Richard King ———————-

We are talking about two different things here.

What I mean by a bracket order is an entry order with linked stop loss

and/or target orders. TWS and the IB servers treat these specially: the

entry order is placed with the exchange, but the stop loss and target orders

are held back until the entry order fills. When it does, they are submitted

to the exchange.

Moreover, if you set transmit=false for all except the last order in the

bracket, then TWS doesn't submit the entry order to the exchange until the

whole set of orders is ready.

The advantages of this are:

1. If your entry order is not filled, then the stop loss and target orders

never get placed with the exchange, so there is no worry about them

executing.

2. It avoids the situation where the entry order executes immediately (eg a

market order or a marketable limit order) and then your connection or your

application breaks before you can submit the stop loss order, leaving you

with an unprotected position.

3. If you cancel the entry order, the stop loss and target orders are also

cancelled automatically.

What you appear to be doing is merely creating an OCA group (for which of

course you have to supply the OCA group name). But this gives you neither of

the benefits of the 'special' bracket order. So if you use a limit or

stop(limit) order to enter, it may never be filled, but there's nothing to

prevent one of your other orders executing (which one would of course depend

on your entry order type).

To create this special order group, you simply have to set the parentId of

the child orders to the orderId of the parent orders: TWS does all the rest

for you.

Note that you can also add the stop loss/target orders after the entry order

has been placed (but before it has executed) using parentId.

By the way, just for completeness, I haven't actually checked recently that

TWS still overwrites any OCA group name you supply when creating a bracket

order in this special way, but it certainly used to. The point is that you

don't need to supply it.

Richard L King

[Q] OCA orders are simulated orders?

[A] by Josh View/Reply Online (#43747)

Added on 07-Feb-2020

OCA groups are a simulated order type not supported natively by exchanges. The orders in an OCA group will be held on the IB server until it thinks one is likely to fill and then it sends that order to the exchange. So you're really only submitting the orders with an OCA group tag to the IB server, not to the exchange. In the case where an earlier order in a group placed milliseconds apart has already filled you would expect later orders in the same group to be rejected, so that is the expected behavior. You can use the Order.ocaType field to choose whether partial fills of an order should cancel or reduce other orders in the group.

[Q] MOC (Market on Close) orders and OCA Group

[A] by ds-avatar View/Reply Online (#48315) 

Added on 18-Dec-2021

MOC orders can be placed in OCA groups in unprotected-reduce mode. That's the single mode that has worked in my experience with MOC orders, with an additional caveat that the closing order will likely not actually be reduced at any stage but only cancelled completely once the other order is completely filled.

[Q] Code: 201 – Order rejected – reason:The OCA group order has already been filled

[A] by Richard L Kins, Josh and others (from this thread)

<tread too long to be copied here.. See online…>

[Q] Order overfilled?

[A] by rwk (from this thread)

I have gotten overfills on USA stocks, though not often.  In my case, it usually happens when I do a cancel & replace.  Sometimes the execution reports are late, and that has been a serious problems lately as I mentioned in an earlier post.

The problem you're describing sounds like a bug in paper trading, assuming you placed only one order and never modified it.

[rwk]

[Q] SHORT selling (SSHORT vs SELL)?

Added on 27-Apr-2015

Hello,

I have been testing my day trading algorithm on the paper account, and I am able to short correctly by simply issuing a "SELL" order. However, in the API manual, it says the order must be one of "BUY" "SELL" or "SSHORT". I was assuming that the "SSHORT" designation was for short selling. However, when I issue that order type in my paper account, I get an error from the API. As far as the paper account goes, using "SELL" works fine for short selling. My question is, will using "SELL" orders for short sales work correctly on the live account? Or do I somehow have to go back to "SSHORT" orders for short sales on my live account? Thanks.

Roy

[A] by Jason Sapp (from this thread)

SELL orders work just fine for US Stocks.  I've been doing it now for approximately 4 years.

Jason

[Q] Changing Order Quantities while order partly filled.

Hi Guys,

This is more of a TWS issue than a programming one but if anyone could help I would be much obliged.

If (say) a limit order is entered for 10 futures contracts and after 6 contracts have been filled I decide that that's in fact all I want how do I change the order to reflect that without actually cancelling the order? From my experience if, after filling the 6, I then amend the Quantity field to 6 it continues to fill up to 10; the same with any amount that I amend the order to lower than the 6 already filled. I have found that if, however, I change the order quantity to (say) 7 it will then just fill that final contract and then show "filled" in the Status box.

The reason I do not want to just cancel the order is that the order is part of a basket that has child orders attached to it and if the initial order is actually cancelled rather than amended then the associated child orders would cancel too – which I do not want.

If anyone could help it would mean a few less grey hairs this end.

Many thanks, John

[A]  by Jsa Jsa

I've figured out a way of doing it which keeps the parent and child orders and works pretty well too; just in case someone has the same problem in the future.

Solution: If the original parent order is to buy 10 contracts and while the order is filling it is decided that (say) 6 is enough, then move the original parent order to a few ticks below the associated child Stop Loss order and change the parent order to just 7 contracts. At the same time also change the amounts of the child orders to (in this case) 6. Then when the child stop or take profit order is executed then cancel the remaining part of the parent order that is left of the original.

I have set up my order entry system to do this automatically but that's what it instructs TWS to do and as I say it works well.

[Q] How do I get commissions from API?

[A] by Brant Hahn

take a look at the openOrder callback on EWrapper – the OrderState object that gets passed in has a property called m_commission on it that will give you a commission value a second or 2 (or more) after your order gets filled.

(update 29-Mar-2018)

This is true, but an orderStatus of filled is not guaranteed. A reliable way to receive commission information is to monitor the commissionReport function, it will have commission information both immediately after the trade and later, in response to reqExecutions. To receive commission information from all API clients it will be necessary to set the API client as the master client.

[Q] Also if I want to see the sum of Commissions in a given day from API ?

[A] by Brant Hahn

you'll have to collect that info yourself from the single trade commissions

[Q] How to check the date when the current position was acquired?

[A] by Kurt

Nothing via the TWS API. You'd have to download statements.

[Q] Can I use just conId for specifying the contract?

[A]

Look at

http://www.interactivebrokers.com/en/p.php?f=programInterface&ib_entity=llc

i.e., the release notes for version 9.64:

"You can now request market data and place orders from the API using the conid, the unique contract identifier. Previously, the conid could only be used for contract details. "

Does this answer your question?

[Q] My order’s place in line!?

[A] by Frank Bell

I've got no special knowledge, but I don't think IB's routing discriminates

between customers. I've had unable orders that should have filled based on the

time they were placed. Naturally I'm unhappy when this happens.

After some email with the SMART group I learned that if, for example, you have

an order to buy 5000 xyz at 5 that's resting on Nasdaq and it becomes possible

to execute 100 on ARCA, the full 5000 will be routed to ARCA in the hope that

the 100 represents the tip of an iceberg order. Of course, once pulled from

Nasdaq, the order has lost its time priority if it's routed back or your price

could be traded through before the order can be routed back. I don't know if

this is the best strategy. I'm believe it's not for very large orders. A

better strategy would seem to be to split the order, routing part to ARCA and

leaving the rest on Nasdaq. But I think that's too much to expect from SMART.

Since there's no guarantee SMART will leave an order undisturbed once initially

routed, there's no way to know where your place in line is.

[Q] Max usage of buying power for DT system

[A]

  • Hi
  • Before running out of account buying power for a DT system, what else
  • conditions should be checked except for buying power itself bigger than
  • zero? I got the following order rejected error msg before running out of
  • buying power. Any suggestions are really appreciated, Thanks,
  • Id: 919 | Error Code: 201 | Error Msg: Order rejected – reason:YOUR ORDER IS
  • NOT ACCEPTED.
  • IN ORDER TO OBTAIN THE DESIRED POSITION YOUR
  • EQUITY WITH LOAN VALUE [95269.89 USD]
  • MUST EXCEED
  • THE INITIAL MARGIN [95902.93 USD]
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Aug 16, 2010
  • View Source
  • I don't think about buying power very much, since I think in terms of
  • margin. But I believe buying power reflects the amount of an ordinary
  • marginable security (e.g. stock) you can buy on margin. So I don't think it
  • applies very well in any more complex margin situation, e.g. involving
  • option strategies.
  • If you are buying stock, then I'd think the simple buying power concept
  • probably applies, and in that case I don't know what would create an
  • exception. But you need to understand margin *completely* regardless. So
  • I'd suggest going into Report Management and producing a margin report and
  • making sure you understand it fully. Unfortunately that is an end-of-day
  • report and may not help you with day trading. In that case you may need to
  • look into the functionality with TWS that may help you analyze your margin
  • situation dynamically. I'm not terribly familiar with those features at
  • this point however.
  • -Kurt
  • Show message history
  • Reply
  • Jason Sapp
  • Aug 16, 2010
  • View Source
  • Hey weidong,
  • I use a fairly complicated position sizing alg, but it starts with an assumption of not risking more than .25% of account equity (including commish and estimated slippage). As i start to exhaust my buying power as the day wears on, i start to reduce subsequent position sizes by a certain percentage. As i exhaust more and more buying power, i continue reducing the size of new positions, until i hit .1%. At that point, i stop reducing buying power for subsequent positions and then just wait until i exhaust buying power. Its taken me a few years to optimize this algorithm, but it works pretty well and i only run out of buying power about once per month.
  • Jason
  • —–Original Message—–
  • From: Weidong Ruan <wdruan@…>
  • Sent: Monday, August 16, 2010 5:18 PM
  • To: TWSAPI@yahoogroups.com
  • Subject: [TWS API] Max usage of buying power for DT system
  • Hi
  • Before running out of account buying power for a DT system, what else
  • conditions should be checked except for buying power itself bigger than
  • zero? I got the following order rejected error msg before running out of
  • buying power. Any suggestions are really appreciated, Thanks,
  • Id: 919 | Error Code: 201 | Error Msg: Order rejected – reason:YOUR ORDER IS
  • NOT ACCEPTED.
  • IN ORDER TO OBTAIN THE DESIRED POSITION YOUR
  • EQUITY WITH LOAN VALUE [95269.89 USD]
  • MUST EXCEED
  • THE INITIAL MARGIN [95902.93 USD]
  • [Non-text portions of this message have been removed]
  • [Non-text portions of this message have been removed]
  • Reply
  • elenosandoval
  • Aug 16, 2010
  • View Source
  • When you subscribe to Account Data, there are about 102 fields of data sent by TWS. Some of them have very unusual names like "Indian Stock Haircut" or "lookAheadAvailableFunds."
  • Useful to your question are fields like "Buying Power" and "Available Funds."
  • Since I trade Futures, of interest to me are also "InitialMarginReq" and "MaintMarginReq."
  • For each contract I trade, my API looks at Available Funds minus my "Cusion" and Initial Margin Requirments for each contract, so I know how many contracts I can trade between, say, 9:30 and 3:45 EST.
  • After 3:45 EST, IB switches to Overnight Margin requirements, which are usually about double the Intraday requirements.
  • Can't help much with stocks and their margin requirements, but, like I said, there are about 102 fields of data, so I'm sure the information you're looking for is in one of them.
  • Good luck,
  • Leno
  • Show message history
  • Reply
  • Weidong
  • Aug 16, 2010
  • View Source
  • The DT sys is for stocks only, works between 9:30AM and 2PM.
  • The sys catched one of 102 fields after subscribing to Account Data, named "BuyingPower", as follows,
  • if(key=="BuyingPower")
  • {
  • m_BuyingPower = atof(val);
  • }
  • Before placing any buying order, sys check if m_BuyingPower minus shares*buyprice is bigger than 5000(always leaving 5000 dollars space), if so, go ahead to place order and minus shares*buyprice from m_BuyingPower. however still got error,
  • Id: 919 | Error Code: 201 | Error Msg: Order rejected – reason:YOUR ORDER IS NOT ACCEPTED.
  • IN ORDER TO OBTAIN THE DESIRED POSITION YOUR
  • EQUITY WITH LOAN VALUE [95269.89 USD]
  • MUST EXCEED
  • THE INITIAL MARGIN [95902.93 USD]
  • Anything wrong with this logic?
  • Thanks a lot,
  • Show message history
  • Reply
  • Kurt Bigler
  • Aug 17, 2010
  • View Source
  • Your logic "sounds fine" given a certain set of assumptions.
  • However I think the real question is how buying power relates to margin.
  • You say you have checked buying power and it is ok, and IB says they have
  • checked margin and it is not ok. So my superficial observation is that you
  • and IB are not talking the same language. I don't know offhand why (for
  • stock purchase) it would not work, except that I recall that some stocks are
  • not marginable, in which case the buying power notion would not apply.
  • Regardless, if you can compute it the way IB computes it (based on EWL and
  • margin) then maybe you will get the result you need, rather than trying to
  • second-guess how exactly how buying power relates to margin and when your
  • model breaks down.
  • But if indeed some stocks you trade are not marginable (penny stocks?) then
  • you would need to have a way to know that. Maybe use your buying power rule
  • for marginable stocks and use simple available funds for non-marginable
  • stocks.
  • -Kurt
  • Show message history
  • Reply
  • Weidong Ruan
  • Message 7 of 7 , Aug 17, 2010
  • View Source
  • Thanks Kurt, Really appreciated your invaluable input.

[Q] openOrder / orderStatus / execDetails sequence?

[A] by Kurt

  • Ron Hinchley
  • Message 1 of 2 , Jul 31, 2010
  • View Source
  • Does openOrder() always come before orderStatus() or execDetails() call backs?
  • Is the 'status' of openOrder() the same as the 'status' of orderStatus()?
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Message 2 of 2 , Jul 31, 2010
  • View Source
  • On 7/31/10 7:22 PM, "Ron Hinchley" <grocthis@…> wrote:
  • > Does openOrder() always come before orderStatus() or execDetails() call backs?
  • >
  • > Is the 'status' of openOrder() the same as the 'status' of orderStatus()?
  • It is not documented that I know of, so if we depend on it we may be
  • disappointed.
  • But yes, that's all true in my experience, re openOrder and orderStatus. I
  • currently use the order status from the earlier message and ignore the other
  • one.
  • Re execDetails I am not sure. It is such a separate entity with its own
  • kinds of unreliability that I don't imagine I would make any assumptions I
  • didn't need to make. For example I would be prepared to handle an
  • unexpected execDetails and likewise the lack of an expected execDetails. So
  • for example after a timeout I would probably request executions, but I
  • haven't done that code yet.
  • I would try to model state in a way that does not depend on the order of
  • execution versus order messages. However I currently plan on assuming
  • paired openOrder and orderStatus messages, unless I discover or hear of any
  • exceptions.
  • -Kurt

[Q] Orders pacing?

[A]

What I remember from a past experiment is if you send orders to the TWS

socket too quickly then TWS will deliberately or otherwise wind up not

accepting an order. It was not related to application processing or

timing. If you send three orders to the socket without delays you will

wind up with orders vanishing. If this has changed then great, but I

wanted to check to see what the current situation is.

From what I remember, and also from Joe's message, TWS does have some

issues if the orders arrive at the socket too quickly.

Measure time to the acknowledgement. If it's a TWS "synthetic" order

emulated by their servers, you get a "pending submit" status if I remember

correctly. If it's sent to the exchange, a "sumitted" status. Typically at

least on a fast exchange like Globex, the delay would be less than 100

msecs. More typically in the range of 60-70 msecs or so for

acknowledgement. I've measured this many times and there is some

variability but that's the speed you can achieve with a properly tuned

system.

In an effort to share some of my trials and tribulations with order placement,

there has always been a problem with IB's order handling (at least since early

2008, up through late 2009) that caused a problem with placing parent/child

orders where a single parent order is placed along with a single set of

bracketed children. If all orders are placed one after another (without any

pause in between them), the problem will manifest itself (at some point).

Last year, I talked to the IB support team for some time about the problem, but

they were never able to fix it or reliably reproduce it. I've written test

programs that create orders just like my application (in a paper trading

account), in rapid succession, but I still have not been able to reproduce the

problem. I've just never been able to properly reproduce the problem in a test

scenario.

The problem did not manifest itself as a pacing violation, but rather, the stop

child order did not get fully committed through TWS and the "T"ransmit button on

TWS was left in an unpressed state. This (of course) seriously sucked, because

my position was unprotected (and it actually burned me a couple times). I ended

up adding in a bunch of code that analyzes all orders to detect if this scenario

actually happens, so that my application can cancel and recreate the

orders appropriately, but with a small delay (mentioned below), the problem

never happens.

I trade US Equities and my system trades an average of about 13 positions per

day (some days less, but some days, I've had more than 40 simultaneous

positions). When I am not using the delay between order placement, the problem

manifests itself about once every 2 or 3 days and I have found absolutely zero

rhyme or reason as to why. The way that I do it and I have never had a single

failure with this technique (with over 9,000 trades) is by creating the parent

order, delaying for 200 milliseconds and then creating the child bracket orders

right away (with—of course—- the final stop order in the bracket being the

one with transmit=true).

Jason Sapp

Here's my two cents:

When I first began coding my API for trading, I built a mimic-TWS order server and found that there were, indeed, some timing issues while communicating over the TCP socket when the order server and my API were running on the same PC.

I had to place time delays here and there in order to get things to work without hangups.

There weren't any problems, however, when I used two computers.

Now, with TWS running on PC-1, for example, and my API and charting programs running on PC-2, when I place a "Basket" order, in which multiple buy/sell orders for DIFFERENT symbols are fired off in rapid succession, without delays, order-execution and open-order data all come back from TWS and everything works fine. No orders are lost.

I do not, however, place other orders, like bracket orders, for the same symbol at the same time, nor do I use the same orderID when placing new orders. I'm not "there" yet.

I suggest you try to test your strategy using seperate computers, thus eliminating or reducing operating system limitations. Also, try fireing off "Basket" orders of different symbols and see if they work, then Bracket orders, etc.

Good luck,

Leno

Yes, I was experiencing the same exact problem as Jason was, where I would

submit a parent order and then a couple of child bracket orders with the last

one being set to transmit=true. I also was placing about 10-15 trades per day

and without a delay of 300ms between the orders, quite often some orders would

show up in TWS, but without the "T" transmitted button pressed.

I would also get a response back through the API that said something like "

Can't find order with ID ="" " I tried as well to set up tests to see if I

could consistently replicate the issue, but to no avail. Even today with the

300 ms delay I will still on the rare occasion encounter this issue.

–Rob Terpilowski

[Q] How to modify the submitted order?

[A1] by Kurt

You simply call placeOrder again with the same Order structure. If the

order id (and client id) are the same it is recognized as a modify to the

existing order.

–Kurt

[A2] by Kurt

You just keep your Order structure around (or all the info needed to

recreate it) and call placeOrder again with the same orderId and it will be

interpreted as a request to modify. Not all fields can be modified. Price

fields generally can be changed, and limit orders changed to market. I'm

pretty sure quantity can also be changed, but I've never done it.

Be warned that modifying an order will typically incur a cancellation fee,

although if the order (or another on the same day) ultimately executes you

may get execution credits against the cancellation fees. But the deal on

execution credits used to be great, and last I looked was almost worthless.

The cancel/modify fee depends on the exchange. Some exchanges charge

nothing for these things, but a directed order will generally have a higher

commission (and possibly poorer execution).

-Kurt

[A3] by  btw12342001 on the bracket order modification:

I was wondering why modifying a leg of a bracket order by changing the price and re-submitting didn't work.  It would trade through the price with no fill.  I noticed in TWS there was an update button and the order status in my log was "pre-submitted".  Obviously the trick is to set transmit=true for those that were false.  I just tested and this works fine.

Maybe obvious, but it vexed me for a while.

[Q] error: The price does not conform to the minimum price variation for this contract

[A] by Ed

ES moves in increments of 0.25 so your stop price must be a multiple of

0.25. For example, setting stop of 1092.60 will cause the error you are

seeing. This commonly happens when the stop price is the result of an

indicator calculation. Rounding to the contract tick amount is typically

the solution.

— also —

[q] A quick question, for emini, does the stop price need to confirm the minimum price variation or just LMT price?

[a] Your order will be rejected if any of the prices are not an exact multiple of the tick size.

[Q] Sometimes reqOpenOrders will not return all open orders. wtf?!

[A] https://groups.io/g/twsapi/topic/4044893#21013

[Q] I have not able to retrieve the previous filled orders through the reqExecutions call

[A] by Kurt

  • Hi there!
  • I have not able to retrieve the previous filled orders through the reqExecutions call. I played with the ExecutionFilter by putting a date string into the m_time and then passed it into the call. Regardless of the parameters I have used, such as setting the client id to "-1", the call would return today's fills. I was not able event to get yesterday's fills. So I suspect that I am doing something wrong here.
  • I wonder if anyone has success in using reqExecutions() call to get previous fills? What value should I use in the ExecutionFilter?
  • Thanks in advance.
  • Reply
  • Kurt Bigler
  • May 26, 2010
  • View Source
  • reqExecutions is apparently filtered by the set of days you have checked in
  • the Trades window in TWS, regardless of what you specify in the API filter.
  • I believe in any case you are limited to the last week.
  • If you are using the Gateway then I think you have no way to specify the
  • filter and I believe someone reported it defaults to the current day only.
  • If it is like TWS in that regard the "current" day might be the day you
  • started it. I have found that with TWS running from the day before I get no
  • executions info until I uncheck yesterday and check today in the Trades
  • window.
  • You can find more details on some of this in the archives from the last
  • couple of months.
  • -Kurt
  • Show message history
  • Reply
  • event_trader
  • May 26, 2010
  • View Source
  • Hi Kurt,
  • Thanks for the quick reply. You are right, if I specify the "All" option in the "trades" page, and leave the page open, I can get all the trades in the "trades" page using the API regardless of the filter parameters. It seems like that the TWS trades page just pass along all the trades in that page to the API call. This is a bad design on the IB API part.
  • I was hoping that the filter can behave more like the flex statement in the account management.
  • Show message history
  • Reply
  • Kurt Bigler
  • Message 4 of 4 , May 26, 2010
  • View Source
  • Thanks, I forgot it was necessary to leave the Trades window open until I
  • tested it just now. I suppose when leaving TWS running overnight and then
  • only getting yesterday's trades it probably means I *had* left the Trades
  • window open. Otherwise perhaps (to be determined) the Trades window will
  • always open with today checked even if it was started on a previous day.
  • Yes, this is definitely a serious problem for people who want to reconcile
  • everything. Also the downloadable statements do not (at least did not used
  • to) use the same form of execution id used by the API, which makes you have
  • to correlate based on the contract description, which I think will be a bit
  • more of a pain programmatically. I haven't approached the problem of
  • reconciling yet because as simple as it should be in concept these little
  • glitches make it a chore.
  • -Kurt

[Q] How to submit a vwap order with start and stop times and other options?

[A] by rfcd453

Since it is not in the current docs, here is how to do it (Java):

Contract m_contract = new Contract();

Order m_order = new Order();

Vector m_algoParams = new Vector();

TagValue m_tagvalue1 = new TagValue();

TagValue m_tagvalue2 = new TagValue();

TagValue m_tagvalue3 = new TagValue();

TagValue m_tagvalue4 = new TagValue();

TagValue m_tagvalue5 = new TagValue();

m_tagvalue1.m_tag = "maxPctVol";

m_tagvalue1.m_value = ".01";

m_tagvalue2.m_tag = "startTime";

m_tagvalue2.m_value = "14:00:00 EST";

m_tagvalue3.m_tag = "endTime";

m_tagvalue3.m_value = "14:30:00 EST";

m_tagvalue4.m_tag = "allowPastEndTime";

m_tagvalue4.m_value = "1";

m_tagvalue5.m_tag = "noTakeLiq";

m_tagvalue5.m_value = "1";

m_algoParams.add(m_tagvalue1);

m_algoParams.add(m_tagvalue2);

m_algoParams.add(m_tagvalue3);

m_algoParams.add(m_tagvalue4);

m_algoParams.add(m_tagvalue5);

m_contract.m_symbol = "AAPL";

m_contract.m_secType = "STK";

m_contract.m_exchange = "SMART";

m_contract.m_currency = "USD";

m_order.m_action = "SELL";

m_order.m_totalQuantity = 100;

m_order.m_orderType = "MKT";

m_order.m_algoStrategy = "Vwap";

m_order.m_algoParams = m_algoParams;

m_client.placeOrder(orderID, m_contract, m_order);’

[Q] Relative Order transmit problem

[A] basically there was no answer, but here’s the port (and definition of Relative Order):

Weidong Ruan

Jul 22, 2010

View Source

Hi,

Relative order sent from my application exists on TWS, but don't transmit.

If clicking T for this order on TWS, it go through without any problem.

Market order, limit order works well with my application. any problem in

following function? any suggestion are appreciated, Thanks.

void CClient2Dlg::placeRelativeOrder(CString symbol, int iOrderID, int

orderquantity)

{

Contract contract;

contract.conId = 1;

contract.symbol = symbol;

contract.exchange="SMART";

contract.secType = "STK";

contract.currency="USD";

Order order;

order.totalQuantity = orderquantity;

order.orderType = "REL";

order.action = "BUY";

order.lmtPrice = 0;

order.auxPrice = 0.01;

order.transmit = true;

m_pClient->placeOrder( iOrderID, contract, order);

}

Relative/Pegged-to-Primary Orders

(quote)

Relative (a.k.a. Pegged-to-Primary) orders provide a means for traders to seek a more aggressive price than the National Best Bid and Offer (NBBO). By acting as liquidity providers, and placing more aggressive bids and offers than the current best bids and offers, traders increase their odds of filling their order. Quotes are automatically adjusted as the markets move, to remain aggressive. For a buy order, your bid is pegged to the NBB by a more aggressive offset, and if the NBB moves up, your bid will also move up. If the NBB moves down, there will be no adjustment because your bid will become even more aggressive and execute. For sales, your offer is pegged to the NBO by a more aggressive offset, and if the NBO moves down, your offer will also move down. If the NBO moves up, there will be no adjustment because your offer will become more aggressive and execute. In addition to the offset, you can define an absolute cap, which works like a limit price, and will prevent your order from being executed above or below a specified level.

If you submit a relative order with a percentage offset, you are instructing us to calculate an order price that is consistent with the offset, but that also complies with applicable tick increments. Therefore we will calculate the order price rounded to the appropriate tick increment (e.g., pennies for a U.S. stock trading at a price over $1.00). Buy orders will be rounded down to the nearest acceptable tick increment and sell orders will be rounded up.

(end of quote)

[Q] How to send post-only order by api?

[A] by Josh View/Reply Online (#44951) 

Added on 28-Jul-2020

Unfortunately no the post-only attribute is only available in TWS.

[Q] How to suppress: Price Management Algo Popup

Every time I submit an order through the API, I get a popup in TWS alerting me that IB has a Price Management Algo that can do something useful if my order is canceled by the exchange. The pop up has option to enable on this order or on all orders of this type for the day. Is there a way to enable this feature through the API so that I don't get a popup every morning? I searched for "Price Management" all across the TWS API docs and I fail to see any reference to it.

[A] by noreply.section@gmail.com View/Reply Online (#44163) 

Added on 03-Oct-2020

There's a boolean field order.usePriceMgmtAlgo that might be relevant in this situation.

— update by alpha pmularien@gmail.com —

This boolean field worked for me. I found that this was the only way to prevent the popup.

[Q] Where do I get the information about how much a order was leveraged? I cannot find a property (for example, leverage) within the "Order" or "Execution" class.

[A] by Josh View/Reply Online (#44307)

Added on 03-Oct-2020

There is leverage reported for an account through the account value 'Leverage-S'. The change in an account's leverage isn't reported along with the order information.

The S (in 'leverage-S') actually stands for 'Securities'. In an IB Universal Account, account values for the securities and commodities segments are reported separately in the TWS Account Window, and in the API.

[A2] by Adam View/Reply Online (#44315) 

The trade execution itself will not be able to tell you what leverage you got. Here are the possibilities:

(1) The best way: you submit a whatif order before you trade it, and you get enough information back (through OrderState) that you can calculate the leverage which you will get if you trade it. You need API 9.76 or later to use WhatIf (that's because in the old version they only provide the 'after' data, thereby causing a race condition – so they fixed that in 9.76). I place a whatif order for 1 share, and then I divide the InitMarginChange by the board lot size to get the leveraged price per share (I then divide that by a currency factor since I trade in multiple currencies, to get leveraged price per share in USD). The leverage is then the current price per share divided by the leveraged price. It's fast enough for my purposes.

(2) If you are only submitting one order at a time, I suppose you can calculate its impact on your total account value after the fact by storing the total account value type 'InitMarginReq' before and comparing to after, but when you place the trade you have to then 'look' for the account value update (only once you see that the trade is is completely executed) to come through and then mark your variable to remember that you've read it once it comes through (so that you don't keep looking for it). But, you need to be able to know that while you're reading and calculating the before and after states that there isn't an execution about to happen due to a different order. If you trade a lot, then you can't do it this way, so use (1) above.

(3) You can use a static table to look it up, but I am told that margin conditions do change sometimes during the middle of the day, so it won't always be accurate – so if that doesn't work for your purposes, then don't do it.

[Q] How can I know when a stop limit order is triggered?

[A] by ds-avatar, Josh, Richart King View/Reply Online (#44556) 

Added on 03-Oct-2020

I believe the order should have generated the Submitted status update message if it triggered. Since there was no such message I'm inclined to believe it did not trigger. Sometimes the stop remains inactive because market conditions technically are not satisfied for it to trigger. This may depend on the value of Order.TriggerMethod field and is more likely to happen in the less liquid stocks especially if the trigger method uses a double check, even when it looks like the stop level was penetrated. My understanding is that the Order.TriggerMethod field works only for IB's own routing (SMART) and actual trigger conditions seem to be pretty complex beyond the brief descriptions in the API docs (look for the page titled "Create a Conditional Order" in TWS user guide for a glimpse on additional technical requirements for some triggers).

I agree with ds-avatar: if you’re not getting a ‘Submitted’ status, then the trigger conditions have not been met.

 

The fact that [child,trigger] changes to [trigger] merely reflects that the parent order is now completely filled, so the child order is no longer a child: it’s just a presubmitted order waiting for its trigger condition to be satisfied.

 

Why the trigger has not been triggered is another matter altogether. It might be worth your while asking IB why it didn’t fire.

 

I notice you have outsideRth=[true], but since I don’t know your timezone I can’t see from your log whether you actually were outside RTH at the time. If you were, then that might be the problem: I recall having had issues with triggers not firing outside RTH, but I’m not sufficiently confident about this to say anything categorical.

 

Richard

Not-native order types, such as stp lmt for US stocks or OCA group orders, are held on the IB servers until their condition is met, and only then submitted to the exchange. So the order status for non-native order types will remain as 'PreSubmitted' even though they are active. (PreSubmitted corresponds to the blue order status color in TWS)

[Q] Order Limitations

[A] by IB

Added on 18-Dec-2021

From TWS API v9.72+: Order Limitations:

Aside from the TWS API's inherent limitation of 50 messages per second implying a maximum of 50 orders per second being sent to the TWS, there are no further API-only limitations. Interactive Brokers however requires its users to monitor their Order Efficiency Ratio (OER) as detailed in the Considerations for Optimizing Order Efficiency IBKB article.

OER = (Order Submissions + Order Revisions + Order Cancellations) / (Executed Orders + 1)

Additionally, please note IB allows up to 15 active orders per contract per side per account.

[Q] How place orders before AND after market hours?

[A] by Josh View/Reply Online (#46634)

Added on 18-Dec-2021

The “Outside Rth” setting in the API applies to both before and after regular trading hours. You just have to make sure you have the “Pre-Open session” flag unchecked in the TWS settings because it’s incompatible.


Combination Orders (aka combo order)

[Q] What are ratios for legs?

can someone explain or has a TWS documentation about what ratios are?

Do I need to know this when constructing legs for a combo-order?

[A] by Despair View/Reply Online (#46818)

Added on 27-Mar-2021

Yes, you need to specify the ratio when creating a combo-contract.

Let's say you want to create a STK/STK combo with the stocks of Amazon and FB and want to buy 3 Facebook stocks for every Amazon stock you sell. You' have to put Amazon in leg 1 with a ratio of 1 and FB in leg 2 (symbols must be paired always in alphabetic order, so FB/AMZN would not work) with ratio 3.

Also you have to specify the ratio as 1/3 and not for example 2/6. Although mathematically  same the second will not be accepted.

If you look at an iron condor the ratio is 1 for every leg (the spread consists of for options and 1 iron condor is 1 option of each leg).

The order of the legs for an option spread should not matter.

The documentation on how to define a spread contract you find here:

https://interactivebrokers.github.io/tws-api/spread_contracts.html

[Q] How would I set the Order::lmtPrice field for a combination order?

[Q] Please explain how to handle combo orders?

[A] View/Reply Online (#45604) 

Added on 15-May-2021

Josh: You have the option of just specifying the price for the order as a whole in the Order class, and not defining a price individually for each leg.

The limit price for the whole combo would just be the sum of the limit prices for each of the legs. So if you wanted to buy 1 leg at $5 and sell another leg at $3, the limit price for the whole combo would be $5 – $3 = $2.

Francois G via groups.io <namasteparis=yahoo.fr (at) groups.io>:

That's the purpose of combo order: dealing with only one order for several legs. When combo order is executed, all legs are executed simultaneously.

Suppose you want to create a combo dealing with 2 products X and Y:

– BUY 6 X at $25

– SELL 3 Y at $10

You should first compute the GCD (Greatest Commom Divisor) of all legs quantities. Here, GCD is 3.

Combo would be created as (+2X -1Y), and order.totalQuantity will be 3. Doing so will prevent you from a poor liquidity market.If you send a combo (+6X-3Y), IB will try to execute all legs at once, but maybe the market only has (+2X-1Y) available. Your (+6X-3Y) combo will never be executed. By simplifying the legs quantities by the GCD, 1 combo could be executed, and you would be happy.

Now, what would be the cost of this simplified combo? When you BUY, you spend money right? So the cost will be  -(2×25) +(1×10) = -40 $

This is a DEBIT combo, When you SPEND money, you want to spend as less as possible right? So you want to BUY the smallest possible price.

Conclusion:you would BUY 3  combo(+2X -1Y) @40   (maybe you will try Order.lmtPrice=39)

Now, suppose you want to create this combo:

– SELL 6 X at $25

– BUY 3 Y at $10

You should first compute the GCD (Greatest Commom Divisor) of all legs quantities. Here, GCD is 3.

Combo would be created as (-2X +1Y),

Now, what would be the cost of this simplified combo? When you BUY, you spend money right? So the cost will be  +(2×25) -(1×10) = +40 $

This is a CREDIT combo, When you EARN money, you want to earn as much as possible right?, so you want to SELL the biggest possible price.

BUT when you sell a combo, it will revert the quantities. If you sell the (-2X +1Y) combo, it will finally BUY 2X and SELL 1Y. So when you intent to SELL a CREDIT combo as expensive as possible, you need to revert the combo first.

Conclusion, you would SELL 3  combo(+2X -1Y) @40   (maybe you will try Order.lmtPrice=41).

Doing so, you avoid dealing with negative combo prices, and being executed the wrong price (wrong side).Trading with negative prices always leads to mistakes.

[Q] Understanding Guaranteed vs. Non-guaranteed Combination Orders

[A] IB website: https://ibkr.info/node/1683

Added on 10-Oct-2015

By Josh View/Reply Online (#45588) 

The option combos native to an exchange are the guaranteed combos. There are also 'non-guaranteed' combos which can be made in many more arbitrary combinations of options, as well as stocks and other instruments. With non-guaranteed combos the (small) risk of having only some of the legs fill is taken by the buyer.

https://ibkr.info/node/1683

https://ibkr.info/node/1323

— also —

nonguaranteed is the only option for many arbitrary combinations where guaranteed combo orders aren't supported. By default combos are guaranteed unless the nonguaranteed tag is specified.. If an order has an excessive number of legs for its type it will just be rejected outright, thats fairly straightforward to test in a paper account.

— also —

There are a huge number of valid guaranteed combos possible at any point in time, so it wouldn't be a simple matter to list them.  Most of them are the well known option combo types (straddle, strangle, vertical spread, calendar spread, etc). The Strategy Builder Tool in TWS is useful for showing different combos available. Or you could always ping the server if you're unsure with a guaranteed combo and it will return an error if its not supported.

[Q] How to create option combo leg without knowing the conId of each leg?

The conId's of each leg are required in order to create an option combo leg, but this requires some lookup (based on the options' expiries/strikes) from the entire option chain. However it is extremely slow to download the contract details of the entire option chain.. Is there a faster way to do this?

[A] by Richard View/Reply Online (#44631)

Added on 08-Feb-2021

Is it that slow?

 

I do something similar for a different purpose: I want all the strikes for a particular expiration above a certain initial strike price, so I have to download them all and filter out the ones I don’t want.

 

Below is an extract of a logfile showing this for MSFT calls. The request for all the calls for the Jun 5 expiry is submitted just after the relevant initial strike price has been determined at 17:24:40.807. The contract details start arriving at or before 17:24:41.596 (the ones with strike less than 172.5 were ignored and not logged), and all of them have been returned and processed by 17:24:41.738. Note that the processing includes requesting and receiving a market data snapshot for each one, so that also happens within that short timespan.

 

That doesn’t seem too bad to me: less than a second.

 

Just make sure you’re not requesting all strikes for all expiries: that really does take a long time. And all strikes for all expiries for all exchanges is even worse.

 

2020-06-03 17:24:40.807 MSFT@SMART/NASDAQ: initial strike price is: 172.5

2020-06-03 17:24:41.596 Contract: MSFT  200605C00175000@SMART; strike: 175; value: 920

2020-06-03 17:24:41.597 Contract: MSFT  200605C00182500@SMART; strike: 182.5; value: 243

2020-06-03 17:24:41.598 Contract: MSFT  200605C00185000@SMART; strike: 185; value: 106

2020-06-03 17:24:41.598 Contract: MSFT  200605C00187500@SMART; strike: 187.5; value: 35

2020-06-03 17:24:41.603 Contract: MSFT  200605C00205000@SMART; strike: 205; value: 1

2020-06-03 17:24:41.603 Contract: MSFT  200605C00190000@SMART; strike: 190; value: 11

2020-06-03 17:24:41.604 Contract: MSFT  200605C00202500@SMART; strike: 202.5; value: 1

2020-06-03 17:24:41.605 Contract: MSFT  200605C00200000@SMART; strike: 200; value: 1

2020-06-03 17:24:41.605 Contract: MSFT  200605C00210000@SMART; strike: 210; value: 1

2020-06-03 17:24:41.606 Contract: MSFT  200605C00180000@SMART; strike: 180; value: 445

2020-06-03 17:24:41.607 Contract: MSFT  200605C00172500@SMART; strike: 172.5; value: 1160

2020-06-03 17:24:41.607 Contract: MSFT  200605C00192500@SMART; strike: 192.5; value: 4

2020-06-03 17:24:41.608 Contract: MSFT  200605C00197500@SMART; strike: 197.5; value: 2

2020-06-03 17:24:41.608 Contract: MSFT  200605C00177500@SMART; strike: 177.5; value: 670

2020-06-03 17:24:41.609 Contract: MSFT  200605C00207500@SMART; strike: 207.5; value: 1

2020-06-03 17:24:41.609 Contract: MSFT  200605C00195000@SMART; strike: 195; value: 2

2020-06-03 17:24:41.610 Contract: MSFT  200605C00215000@SMART; strike: 215; value: 1

2020-06-03 17:24:41.738 Contract: MSFT  200605C00220000@SMART; strike: 220; value: 1

2020-06-03 17:24:41.738 Target contract: MSFT  200605C00175000@SMART; strike: 175; value: 920

 

Richard

[Q] Non-Guaranteed Combination Orders

[A] IB website: https://ibkr.info/node/1323

Bracket orders

[Q] What is "Bracket Order"?

[A] by IB website:

Definition of the bracket order

https://www.interactivebrokers.com/en/index.php?f=583

API Docs on bracket order

https://interactivebrokers.github.io/tws-api/bracket_order.html

[Q] Note on the Bracket Order child order allowed types?

[A] By Yair Altman View/Reply Online (#46372)

Added on 04-Jan-2021

Note that when you attach multiple child orders, IB may place limitations on the possible combinations of allowed child-order parameters. For example, when you add a MOC exit order to the standard bracket child-orders, IB only allows the child orders to have an OCA type of 3 (Reduce on fill without block) and not 1 (Cancel on fill with block) or 2 (Reduce on fill with block). Without the extra MOC child order, you could have your child bracket orders also use OCA Types 2 & 3, not just 1. Such limitations may depend on your specific secType and exchange.

[Q] Bracket Order Quantity is capped? (yes – by parent's qty)

[A] View/Reply Online (#45913)

Added on 03-Dec-2020

You are correct in that the order quantity for any child order is fixed to the quantity of the parent order. You can't change that.

I guess that you will need to change your orders such that the last take profit and stop loss orders are not linked to any parent order. I'm not sure if you can do that in TWS (having the orders reside at either IB or the exchange), or that you will need to write code to continuously monitor your position size and submit orders based off of that.

— also —

In case of a bracket order must the child orders have the same quantity as the parent order. You can't modify the quantity of the child order(s) independently of that of the parent order.

[Q] Margin for generic orders basket, multiple leg

[A] View/Reply Online (#45854)

Added on 03-Dec-2020

TLTR, but QuanTradder mentining there that currency trades or CFD, are not supported in combos at all! Hm… interesting!

[Q] Bracket order: Why Sell and Stop don’t cancel each other?

Hi,

I'm trying to place a bracket order (Buy, Sell LMT, STP) using the Java IB

API but the Sell and STP don't cancel each other.

I'm using the Buy order ID as the parent ID for both the SELL and STP

orders.

Any ideas what the problem is?

Thanx,

Rebecca.

Hi,

Here's the code:

Order order = createOrder("BUY", 1, "MKT");

Contract buyContract = createContract("ES", "FUT", "GLOBEX", "USD","20100618",null,0.0);

eClientSocket.placeOrder(orderId++, buyContract, order);

lastOrderId = orderId;

           

Order order2 = createExOrder("SELL", 1, "STP", lastOrderId, 0.0, currentPrice-gain, false);

Order order3 = createExOrder("SELL", 1, "LMT", lastOrderId, currentPrice+loss, 0.0, true);

           

eClientSocket.placeOrder(this.orderId++, buyContract, order2);

eClientSocket.placeOrder(this.orderId++, buyContract, order3);

Where the methods are defined below:

        protected Order createExOrder(String action, int quantity, String orderType, int parentId, double lmtPrice, double auxPrice, boolean transmit)

        {

           Order order = super.createOrder(action, quantity, orderType);

            order.m_parentId = parentId;

            order.m_lmtPrice = lmtPrice;

            order.m_auxPrice = auxPrice;

            order.m_transmit = transmit;

            return order;

        }

        protected Order createOrder(String action, int quantity, String orderType) {

            Order order = new Order();

            order.m_action = action;

            order.m_totalQuantity = quantity;

            order.m_orderType = orderType;

            // order.m_transmit = true;

            return order;

        }

        protected Contract createContract(String symbol, String securityType, String exchange, String currency) {

            return createContract(symbol, securityType, exchange, currency, null, null, 0.0);

        }

   

protected Contract createContract(String symbol, String securityType, String exchange, String currency, String expiry, String right, double strike) {

            Contract contract = new Contract();

            contract.m_symbol = symbol;

            contract.m_secType = securityType;

            contract.m_exchange = exchange;

            contract.m_currency = currency;

            if (expiry != null) {

                contract.m_expiry = expiry;

            }

            if (strike != 0.0) {

                contract.m_strike = strike;

            }

            if (right != null) {

                contract.m_right = right;

            }

            return contract;

        }

The result is that the Future was bought, and I had 2 open orders one SELL LMT and one SELL STP, that both seemed to have been transmitted.

Later on the STP caught and was executed, but the SELL LMT Order was not automatically cancelled.

Any ideas what I'm doing wrong?

Thanx in advance for the help,

Rebecca

[A1] by Jiang LingyuMay 31, 2010

you can set the same group name for your LMT and STP orders. This will cause OCO

[Rebecca]

I tried that, but it didn't seem to have any affect.

Can you give an example?

[A2] by Richard King

At a glance, I'd say that the problem is with your first call to placeOrder incrementing orderId. The increment happens after the current value is passed as the argument to placeOrder. Consequently lastOrderId is one greater than your entry order's id, and therefore the parentId of the child orders is not correctly set.

Richard

[A3] by Jiang Lingyu

From: TWSAPI@yahoogroups.com [mailto:TWSAPI@yahoogroups.com] On Behalf Of Jiang Lingyu

Sent: 01 June 2010 09:36

To: TWSAPI@yahoogroups.com

Subject: Re: [TWS API] bracket orders

Something like this:

order2.m_ocaGroup = "group1";

order3.m_ocaGroup = "group1";

hope it helps.

[A4] by Richard L King

Actually, TWS automatically assigns an OCA group to orders that have parentId set.

In fact it ignores anything you specify as in your example. (At least that used to be the case – I haven't tested this with recent versions.)

accepted answer

[Rebecca]

Thanx, that solved the problem.

I was looking for wrong usages of the API, instead of regular programming bugs 🙂

I had another issue.

The STP was not transmitted even though the last order, the LMT sets transmit to true.

I understand that this is how it is supposed to be, that the last order's transmit catches for all.

It only worked once I set the transmit flag to true for all orders.

[note] by scourt2000

Also, on a market order, your distance from the two bracketing orders you setup will be off by a tick or two since you're entering them at the same time as your entry, especially with a market entry order. If you use a limit order, then the worst that will happen is that, in a volatile market, you'll get a better entry by a a tick and then you'll have a tighter stop-out than you originally wanted.

What I do is submit either a market or limit order and then I submit the bracket once I know the entry price as reported by IB.

[note]

All I know is that the brackets should have a matching, OCA

string. Another bracket needs a different string.

[note]

Let me spell it out again. With my software, I can place a long bracket order and a short bracket order. Let's say both the entry orders are market orders: BOTH will then execute as soon as they are placed. So as far as the IB account is concerned, the net position is now flat. But that has absolutely no effect on the two stop loss orders and the two target orders, which remain in the system (as two separate OCA groups) and will execute (and cancel the respective other halfs) when/if the relevant prices are hit.

If you have tried this and it didn't work, then you have simply done something wrong. You cannot tell me that it doesn't work, because it does!

Richard

[A5] by Ed (from other thread)

You need to set Transmit to False for all orders except the last one.

Send the Parent order first, then the children. On the last child order

set Transmit to True and the whole lot will be submitted.

— also good note from Ed —

I'm not sure what you are trying to achieve, but there are at least

two ideas:

1) Use the standard parent/child setup and adjust the child prices after

the parent executes

2) Submit a normal order for entry and then use an OCA group for the

profit/stop orders after the entry executes

[Q] Bracket orders: 3 or more childs? grand child? stop order as an entry?

I am newbie and wish you guys give me some advise.

Searched from API Reference Guide, I know there are 3 orders (one parent and two children) in a bracket order,

entry order (parent), market or limit, transmit = false

  stop order (child1), stop, transmit = false

  target order (child2), limit, transmit = true

Can we use stop order as an entry order?

Can we only have one child? Is it possible to add 3rd child order or more?

Is it possible to add a grand child order whose parent is child1?

Thanks

[A1] by Richard L King (from this thread)

My advice to you would be to find out answers to this sort of question yourself.

The amount of code needed to produce a small test app is not great, and producing it would be a good exercise for you. Once you have a basic working framework (connecting, logging errors, etc) you can easily clone it for new test apps in a few seconds.

Alternatively you could use the sample apps provided by IB to try these out – I very much recommend that you learn how to use one of these programs, because they can save you hours of time and you get the correct answer, whereas posting here you have a potentially long delay before you receive an answer, and there is no guarantee that any answer provided will be correct.

Having said that, I can tell you that:

– You can use a stop order for entry

– As far as I’m aware, you can use any number of child orders. Make sure you set transmit-false on all except the last to be placed. Note also that you can add child orders after the initial bracket order (or single order) has been placed

– I don’t know if you can use a child order as a parent, but I don’t see why not

But I urge you not to just believe me about any of this: try it yourself!

Richard

[A2] by orionn2 (from this same thread)

The answer to all your four questions is: yes, but to avoid problems place the stop loss last with transmit = true; parent and target should be placed before the stop loss order with transmit = false (this is to avoid entering a position without the stop loss order that may fail to be placed/transmitted due to an error – if the stop loss cannot be placed for some reason then the other orders will not be transmitted to the exchange).

[Q] Bracket orders: edge cases

For anyone using bracket orders(just saw another email asking about coding

example), could you answer my questions below?

1. If the parent order is a limit order and got partially filled, will the

two children still be activated? Or only when the parent got fully filled?

2. The two children work as OCA orders, right? Still about partial fill, if

one child got partially filled, will IB adjust the quality of another?

3. If the parent got canceled, will IB cancel children automatically? What

if the parent got partially filled then canceled?

Since I am using limit entry orders, the partial fill happens a lot, I am

wondering whether to use bracket orders could cover all the edge cases? If

not, I would rather handle by my own.

Thanks!

Jian

[A] by Christian Gross (from this tread)

It has been a little while since I did brackets. But here goes, anybody feel free to correct me.

1) yes

2) depends, by default yes, but check stock (assuming stock) OCA Group presets

3) yes the children will get cancelled. Assuming you did the cancelling. THOUGH there are exceptions to the rule. If there is a partial fill, and you cancelled just as the order was filled then your cancel will not be listened to. You get an error of non-existent order.

[Q] Bracket orders: How to place bracket order via API (recap)

Added on 28-Apr-2015

[A] by Richard L King (found here)

How to create a bracket order using the API has been discussed many times

here, and it certainly is possible (and very easy).

Here's how to do it:

1. Create your entry order, set transmit to false, and place the order.

2. Create your stop loss order, set its parentId to the entry order's order

id, keep transmit set to false, and place the order.

3. Create your target order, set its parentId to the entry order's order id,

set transmit to true, and place the order.

The transmit settings ensure that none of the orders are sent from TWS to

the IB servers until all the orders have been set up – then they are all

sent together. You don't have to do this, but if you don't you face the

potential problem of the entry order being filled before the stop loss and

target orders have been sent.

You don't have to have both a stop loss order and a target order. And if you

do have both, it doesn't matter what order you create them in (but of course

you must create and place the entry order first).

Don't be tempted to set the OCA group on the stop loss and target orders: it

isn't necessary, because TWS does that automatically and in fact it will

ignore anything you set.

You can modify the members of a bracket individually in the same way as any

other order.

Richard

[Q] How to modify pricing on a bracket order?

[A] by Josh (found here)

Added on 10-May-2017

It would be very similar for a bracket order. You just call placeOrder individually for the order(s) in the bracket you'd like to modify.


Placing Orders (stocks)

[Q] How can I keep a reference to these manual trades when they pop up in openOrders if all the IDs are 0 at the time?

[A] by Josh View/Reply Online (#40989)

Added on 01-Nov-2018

You can use the Order Reference field to manually label orders. For instance if you right-click on a tab in TWS Classic View and choose Settings, an Order Reference field can be entered which will be the default for all orders created on that tab. This string is then accessible to the API in Order.OrderRef.

[Q] What "Order Inactive" really means?

[A] 

  • When I try to place an order for an ES future, I'm getting "Inactive" from openOrder and from orderStatus, then an error message that the order was rejected. But I never get an order status of "Cancelled".
  • Regardless of the reason of the rejection which is not the issue here, is it true that I cannot determine from openOrder and orderStatus alone whether the order has been definitely rejected or if it just has been inhibited for some reason and could possibly be accepted at a later time?
  • PS: I already placed this question in the IB forum, and got no answer.
  • Reply
  • Richard L King
  • Aug 13, 2010
  • View Source
  • You get this when there is something wrong about the order that could be put
  • right. It gives you the opportunity to fix it and then resubmit it.
  • Of course an automated program is probably not going to be able to fix it,
  • or it wouldn't have submitted it incorrectly in the first place, so it will
  • probably want to cancel the order. But the API is also used by many manual
  • trading clients, so it may be that the user has specified something wrong on
  • the order and can perhaps put it right.
  • 'Inactive' means exactly that: the order will sit there indefinitely in TWS
  • awaiting further instructions. It won't somehow become active again without
  • user or API-client intervention.
  • From: TWSAPI@yahoogroups.com [mailto:TWSAPI@yahoogroups.com] On Behalf Of
  • Alexander
  • Sent: 13 August 2010 12:09
  • To: TWSAPI@yahoogroups.com
  • Subject: [TWS API] Order Rejected not an order status ?
  • When I try to place an order for an ES future, I'm getting "Inactive" from
  • openOrder and from orderStatus, then an error message that the order was
  • rejected. But I never get an order status of "Cancelled".
  • Regardless of the reason of the rejection which is not the issue here, is it
  • true that I cannot determine from openOrder and orderStatus alone whether
  • the order has been definitely rejected or if it just has been inhibited for
  • some reason and could possibly be accepted at a later time?
  • PS: I already placed this question in the IB forum, and got no answer.
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Aug 13, 2010
  • View Source
  • Contrary to what Richard said about this, I have found that in some
  • situations orders will be "Inactive" after an error, but you are *not* able
  • to modify and resubmit. I believe you get a duplicate order id (not sure)
  • if you attempt it.
  • Indeed as you said, there is absolutely no way to know whether "Inactive"
  • means the one thing (correct and resubmit) or other (create a new order from
  • scratch). It is a clear bug, and I have just never bothered to report it.
  • I believe what I started working on (I forget the status of that
  • development) is to make distinctions based on the error code and to correct
  • the order status.
  • (Sometimes I just get discouraged about reporting bugs to IB and just try to
  • find my own fix. This seems to be a fairly common attitude that develops
  • for people, from what I've been hearing on this list.)
  • There is yet another kind of problem that can come up. An order can be made
  • Inactive due to an error, and resubmitting is accepted, but correcting the
  • erroneous field does not clear the error. This is the case for setting (or
  • failing to set) the account field for a single-account order in a FA
  • scenario. Apparently IB's code reads the account field on the initial call
  • to placeOrder and does not update it on subseuqent placeOrder calls. So if
  • the account code was omitted it can not be added later and the order must be
  • scrapped.
  • -Kurt
  • Show message history
  • Reply
  • Alexander
  • Aug 16, 2010
  • View Source
  • OK, Order Rejected apparently does not exist as a order status and we don't know exactly what "Order Inactive" really means.
  • This is what I have recieved upon issueing a ES Sep 2010 order without sufficient margin, in this chronological order:
  • openOrder(…) with status Inactive
  • orderStatus(…) with status Inactive
  • error(…) 204 order rejected
  • openOrder(…) with status Inactive
  • orderStatus(…) with status Inactive
  • The order does not hang in TWS but does not appear at all. (Why openOrder and orderStatus two times?)
  • This I have tested on a F&A simulation subaccount, because institutional multiaccounts are not available as simulation accounts.
  • For the time being, assuming that my program does not issue orders with missing mandatory information, I will treat orders returning a status of Inactive as rejected and virtually nonexistent.
  • Thank you Kurt and Richard for your inputs.
  • Alexander
  • Reply
  • Kurt Bigler
  • FWIW, taking a look at my code, I might as well complete the info I previously started. I have not yet seen error 204. I have 201 listed as being prefixed
  • Aug 16, 2010
  • Alexander
  • Aug 22, 2010
  • View Source
  • Error 204 was a typo. Should be 201. The reason for the rejection was insufficient buying power. And no, I would never try to resend the order with the same orderId, I'm always using a new orderId. The red order on TWS will disappear sooner or later.
  • It is good to know that an order which has status inactive will never activated itself at a later time, as Richard pointed out, so I can treat it as rejected.
  • If anyone ever had an order status of inactive WITHOUT an accompagning error 201 I would like to know of it.
  • Alexander
  • Show message history
  • Reply
  • Kurt Bigler
  • Aug 22, 2010
  • View Source
  • I definitely had inactive without an error 201. For me the "Rejected" which
  • resulted in "Inactive" rather than "Cancelled" was new and anomalous
  • unusual-case behavior that broke code previously working written a couple
  • years earlier.
  • 201 is specifically "rejected". I have had orders remain inactive due to
  • some error or warning that did not say "rejected". (But also I think IB
  • keeps changing this kind of behavior.) I'm pretty sure every order I
  • submitted with transmit false used to produce an order status message
  • "Inactive". But if I remember right such orders no longer result in any
  • order status message at all, so I recoded my app to set my own order status
  • code "Local" for this situation in which only the API app seems to know
  • about the order. In short it used to be the case that "Inactive"
  • corresponded to orders displayed as "Gray" in TWS, which included orders
  • never transmitted and orders that were "soft" rejected. I used to get order
  • status messages for these in all cases (including untransmitted), and now I
  • no longer do in some cases. It is sort of a pain in the ass because it
  • means I have no feedback from TWS that it has received my untransmitted
  • order. So if I send it again with transmit=true I have no sense in advance,
  • like I used to, that all the order fields were already checked and the
  • transmit would fly.
  • So this behavior sucks but I'm so disgusted with the garbage IB produces
  • that I just sag slightly further into resigned depression instead of flying
  • into a rage like I used to.
  • -Kurt
  • Show message history
  • Reply
  • Alexander
  • Message 8 of 8 , Sep 2, 2010
  • View Source
  • Reply from IB's API support:
  • Hello Alexander,
  • You are right . When order has been rejected, OrderStatus event would show "Inactive" status and Error event delivers rejection message with the detail description. I strongly recommend that you monitor both OrderStatus and Error events to determine if the order has been placed successfully or not.
  • Regards,
  • Raymund
  • IB API Support

[Q] How to send a limit order then if the order is unfilled after a certain amount of time have the order be cancelled and do market order?

Basically can this be done in just one API call or do I have to cancel the order in code and then resend order as market order ? (Q On Mon, Nov 18, 2013 at 6:38 PM, <qjohnny2000@yahoo.com> wrote:)

[A1] (A by Brant Hahn brhahn@gmail.com via yahoogroups.com )

on your initial limit order, set the following value:

m_goodTillDate;  // FORMAT: 20060505 08:00:00 {time zone}

then when you get the cancelled callback from TWS, submit a new market order

[A2] Rob Terpilowski <rterp99@yahoo.com>

You may also be able to change the order type from LMT to MKT after your time threshold has expired.

[Q] order.m_account = null ?!

by orionn2 <no_reply@yahoogroups.com>:

has anyone ever experienced the situation of order.m_account field being null when orders are received in the openOrder method? This issue is not allowing filtering of orders by sub-account. looking for your feedback.

[A] by Kurt Bigler kkb@breathsense.com

I see that in C++ the field is being decoded by the default OPEN_ORDER processing (for version 9.67), so you should be good in that regard.

However for various reasons I decided it would be a good idea to store/archive my own order information.  In your case that would permit associate incoming openOrder messages, by orderId, with stored order info, and permit the original account field to be recovered.

I'm not sure if it should be necessary in your case.  I discovered reasons for doing it related to the fact that orders can be sent to TWS with placeOrder false, and such orders are not reported back by reqOpenOrder, etc., thus orphaning them from API access if my API app quits without saving its own order state and restoring it on startup.  Some things related to this also got worse at one point (in one of IB's improvements), forcing the issue.

-Kurt

[A2] by vanx23@outlook.com

This sounds suspicious to me.

I have two individual accounts. The field 'AccountID' is never NULL or empty for each OpenOrder messages my application process.

[A3] by orionn2 <no_reply@yahoogroups.com>

Just to clarify that this is happening on an Advisor account (not on individual accounts) when an order is being created and automatically split/allocated to three sub-accounts using the latest TWS version.

[Q] m_shortSaleSlot's field?

[A] by Kurt

The comment in C++ says:

"1 if you hold the shares, 2 if they will be delivered from elsewhere.

Only for Action="SSHORT"

And I believe (but can not find the doc saying it) that SSHORT is

institutional only. It makes sense anyway.

This does not specifically say what you should set the field to if it

doesn't apply. In such cases with TWS API, I let the default initialization

of field determine the value to use. shortSaleSlot is defaulted to 0 by the

C++ constructor. So just don't set the field if it doesn't apply, and the

default should be right.

-Kurt


Placing Orders (options)

[Q] What is the IV in options model used by IB?

[A] View/Reply Online

Added on 25-Jan-2021

Switch over to TWS API doc, that always helps to understand low level details 😉

https://interactivebrokers.github.io/tws-api/tick_types.html

Basically there are 4 types of option greeks + IV around on TWS:

– Greeks calculated on Ask Price (tick id 10)

– Greeks calculated on Bid Price (tick id 11)

– Greeks calculated on Last Price (tick id 12)

– Greeks calculated on Model Price (tick id 13)  

Model does use mid-price (?? not 100% sure about that), HV, ect. of underlying and black-scholes by default. Unless you modify it to create your own option pricing model ( https://www.interactivebrokers.com/en/index.php?f=14278 / https://www.interactivebrokers.com/php/whiteLabel/Interactive_Analytics/modelEditor/intro.htm ). But I have not done that so far, so can't help, if you want to model your own pricing formula inside TWS 😉

[Q] How to get Option BID/ASK and GREEKS after the market close?

[A] by Josh View/Reply Online (#47362) 

Added on 18-Dec-2021

To receive the last available data from the markets after they have closed you need to switch to ‘frozen’ data.

https://interactivebrokers.github.io/tws-api/market_data_type.html

with reqMarketDataType(2)

[Q] How to get daily volume for options?

Options don't have end of day data so the best I could come up with was getting large bars (like 8 hours) and using those to see if there was volume?

[A] by  Josh View/Reply Online (#40025)

Added on 10-Jul-2018

That should work fine. There isn't really any other way to do it.

[Q] See also: How to request contract details in order to get the options for a specific future?

This Q/A was moved into "Futures" section here:

[Q] How to request contract details in order to get the options for a specific future?

Added on 12-Jun-2022

[Q] How to Monitor Stock Loan Availability?

[A] by  Jack Jost jqjost@gmail.com View/Reply Online (#39012) 

Added on 27-Nov-2017

(quote from https://ibkr.info/article/2024 )

Monitoring Stock Loan Availability

Overview:

IB provides a variety of methods to assist account holders engaged in short selling with monitoring inventory levels and borrow costs/rebates. The level of detail available, the time frame covered and the manner in which the information is accessed vary by method and a brief overview of each is provided below.

Public Website

Interested parties may query the public website for stock loan data with no user name or password required. To start, click here and select the country in which the stock is listed. If the number of available issues exceeds that which can be reasonably presented on a single page, results will be organized by symbol in groups, with hypertext links allowing further drill-down. A quick search box allowing direct query for a given symbol is also provided. Query results include the product description, currency of denomination and a link titled “Check Availability” which displays the quantity of shares available to borrow.

 Public FTP

The public FTP site also requires no user name or password to access and provides stock borrow data in bulk form via a pipe delimited text file. The URL necessary to request files varies by browser type as outlined below:

1.        Mozilla Firefox

ftp://shortstock: @ftp3.interactivebrokers.com (Note there is a space between shortstock: and @ which represents the password).

You will get a message stating that you are about to connect to a website that does not require authentication. Select "Yes"

2.        Internet Explorer

ftp://shortstock: @ftp3.interactivebrokers.com (Note there is a space between shortstock: and @ which represents the password).

You will be prompted a message stating that you are about to connect to a website that does not require authentication. Select "Yes"

3.        Google Chrome

ftp://ftp3.interactivebrokers.com

Username:shortstock

Password: blank (n/a)

Within the site, individual files will be organized by country of listing with checkboxes provided to specify those desired which can then be downloaded into a single file by selecting the Submit button.

 

 

Outlined below is a snapshot of the sample file output which includes the stock symbol, currency of denomination, name, contract identifiers (IB’s and the ISIN), rebate & fee rates and shares available. This file may be also imported into applications such as Excel for sorting, filtering and analytical purposes.

 

Stock Loan Borrow (SLB) Tool

The SLB tool is made available to IB account holders through login to Account Management (Support then Tools menu options). This tool allows one to query information on a single stock as well as at a bulk level. Single stock searches can be performed by symbol/exchange, ISIN or CUSIP numbers. At the single security level, query results include the quantity available, number of lenders and indicative rebate rate (which if negative, infers a borrowing cost expressed as an annual percentage rate and, if positive, the interest rebate paid on cash proceeds securing the loan in excess of the minimum threshold). Information regarding the quantity of shares available to borrow throughout the day for the most current and past half hour increments is also made available.

 

In addition, borrowers interested in the trend of rates over the prior 10 day period can view the minimum, maximum and mean rates for each day.

This tool also allows one to upload a text file (with symbol/exchange or ISIN detail) and search for availability of multiple stocks in bulk within a single query. These bulk requests will then generate a .CSV file similar to the sample file output made available through the public FTP site.

Related Articles

(/quote)

[Q] How to find SLB info?

I'd like to get the SLB (Short stocks Loan / Borrow) fee rate via API.

I can't find it anywhere (tick data, fundamental data reports from Reuters…).

Anybody knows if it is possible?

[A] by mattpearson911@yahoo.com

Added on 07-Oct-2016

You can download them here:

ftp://ftp3.interactivebrokers.com/

login: shortstock

No password (leave blank)

[Q] TWS and Option Spreads

[A] read full  thread

[Q] weird theta

Has anyone seen a very large theta returned by tws api? I got the theta 1.7976931348623157E+308 for tsla 195C. It must be an error at tws server side, right? btw, I got all other greeks right.

[A] by Kurt

Are you sure that isn't the max double value used to mean undefined or no data?  It is a very familiar looking number, amazing how many times it comes up in questions on this forum.  Wikipedia shows:

    7fef ffff ffff ffff16   = (1 + (1 – 2-52)) × 21023 

                        ≈ 1.7976931348623157 × 10308 (Max Double)

IB is inconsistent in quotes as to what value is used for the no data case.  Sometimes it is –1, sometimes -.99, sometimes 0, sometimes the "maximum" value.

At the moment my app is displaying the undefined value for many TSLA calls expiring today.  My code converts the special values to something displayable.  I checked a couple other expirys and they have normal values.

IB's strategy is for the greek to be undefined if the calculations are not stable, which in the case of theta is not surprising on expiry day.

-Kurt

— also —

Ding,

[These comments apply to C++.]

The value in question is DBL_MAX, or that is one way you can refer to it in your code, and is one way IB refers to it in their socket interface layer code (e.g. in EClientSocketBase::EncodeFieldMax), although they also use

    #define UNSET_DOUBLE DBL_MAX

in some cases (e.g., EClientSocketBase::DecodeFieldMax, demonstrating some inconsistency).

It is not at all weird for the API to return DBL_MAX for a greek.  You will get that value all the time and you will need to check for DBL_MAX if you are using the greek values from the API.  I just checked my code and it is indeed DBL_MAX that I check for and interpret in the usual way (for the TWS API) as meaning no value is available.  I check for DBL_MAX in the case of the following arguments to the tickOptionComputation callback.

    impliedVol

    delta

    gamma

    vega

    theta

    optPrice

It may be true that these values occur in the API even when TWS is still displaying a value.  It might just be that TWS is a lot less fussy about displaying something (valid or not) whereas IB may not want to risk sending an unstable value to the API when it is conceivable that someone might conceivably trade automatically on it.

Come to think of it my recollection is that greeks are always displayed in the option trader.  I could be wrong but don't remember the columns ever being sparsely blank.  Meanwhile I am used to seeing blank greek columns in my API app, quite often (based on the occurrence of DBL_MAX), intermixed with other greeks for which values are provided.

I see that my code also checks UNSET_DOUBLE (same value as DBL_MAX) for commission and various order fields.  This value (or UNSET_INTEGER = INT_MAX in the case of integers) is used in a lot of places.

It had been my impression that this value may be used for almost any tick (quote) field but at the moment I am not finding evidence of that.  For TICK_PRICE/tickPrice, DECODE_FIELD is used, not DECODE_FIELD_MAX.  DECODE_FIELD_MAX appears to be used for OPEN_ORDER only.  DECODE_FIELD_MAX is actually only used to permit IB to send the null string on the socket and have it mean DBL_MAX.  DBL_MAX can always be sent on the socket directly, and apparently that is what is done for the greeks since DECODE_FIELD, not  DECODE_FIELD_MAX is used there.

However I see no signs of my code special-casing or DBL_MAX or UNSET_DOUBLE in my code except as mentioned above.

-Kurt

[Q] How should an API client detect an option exercise, the exercise being done manually in TWS?

Hi,

How should an API client detect an option exercise, the exercise being done manually in TWS?

The API client receives an open order event, but annoyingly enough, the order event is always  a "buy" instead of the more logical "sell" (an option exercise can only be done on a long option; closing a long position is better characterized as a sell than a buy, in my mind):

5;34;-279;283326534;DIA;OPT;20171020;231;C;100;SMART;USD;DIA   171020C00231000;DIA;BUY;100;LMT;0.0;0.0;DAY;;U###;C;0;;0;1689659455;0;0;0;;1689659455.0/U###/100;;;;;;;;;;0;;-1;0;;;;;;;0;0;0;;3;0;0;;0;0;;0;None;;0;;;;?;0;0;;0;0;;;;;;0;0;0;;;;;0;;IB;0;0;;0;0;Inactive;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;;;;;;0;0;0;None;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;1.7976931348623157E308;0;;;;1.7976931348623157E308

If the order event was a sell, then it would be easy to handle just like any other order event received from TWS.

Alternatively, if the order event was somehow identified as an exercise, then I could swap the buy to a sell.

But as it is, I can't see anything in the order event that shows it as an exercise, and I can't just go swapping buy to sell in all order events I receive…

Thanks!

Jimmy

[A] by Josh: View/Reply Online (#38869)

Jimmy,

I agree it's not the most intuitive, but option exercise instructions will always appear with order side = "BUY" to orderStatus. The order status behavior is also different because there will not be an orderStatus returned after calling reqOpenOrder or reqAllOpenOrder, only a warning message.

The exercise request can be identified by the '0' limit price, since this is not possible for any other orders not involving a combo contract.

Josh

[Q] How to find out available strike prices for options ?

happen to be the same as the next question:

[Q] How to get all valid options (scrape option chains)?

[A] (using python) by Juergen juxeiier@gmail.com View/Reply Online (#46782)

Added on 20-Mar-2021

Hi,

I have created a little library which can scrape option chains in a (hopefully) easy manner.

The topic is discussed here

https://groups.io/g/twsapi/message/46566

https://groups.io/g/twsapi/message/42241

https://groups.io/g/twsapi/topic/81356218

The source code, together with installation details, is here https://github.com/juxeii/ibtools.

The usage is then as simple as

chain = ibt.getOptionContractsUpUntilDays(aapl, 60)

or

aapl=Stock(symbol='AAPL', exchange='SMART/AMEX')

chains = ibt.getOptionContractsUpUntilDays(aapl, 60)

It works on my site, but is very new, so probably some shortcomings 🙂

Would be great if someone could test it.

Cheers,

Juergen

[A2] (using ruby) by Hartmut Bischoff

Added on 20-Mar-2021

Hi,

ib-ruby (https://github.com/ib-ruby) has the ability to work on atm, itm and otm-option-chains.

documentation

https://ib-ruby.github.io/ib-doc/atm_options.html

implementation

https://github.com/ib-ruby/ib-extensions/blob/master/lib/ib/option-chain.rb

Perhaps its worth to take a look, to complete your program

[Q] How to pull the entire option chain?

[A] by Kurt (from this thread)

There is no way to get *just* the strikes. To get all possible strikes

across all possible expiry's you have to poll the entire option chain and

take note of what you want and throw the rest away.

To do this you use reqContractDetails with an ambiguous contract

specification, i.e. out of strike, expiry, and right, you simply don't *set*

the fields you want wildcarded. Then you get back a whole series of

contractDetails responses from the single request, which you can track by

the request id. You have received the whole chain when the

contractDetailsEnd message arrives for that request id.

You can by this method also specify an expiry and right ("C" or "P") and get

all strikes for that expiry and right. Any combination you like. But there

is no independent list of expiry's and no independent list of strikes, only

the combinations. If you leave the right unset ("") you will get both calls

and puts.

When last tested, you can also partially wildcard the date, e.g. by

specifying only the beginning portion of the date, including just the year,

or just the year and the month (e.g. "201201") rather than specifying the

full date code. This is less convenient now with weekly expiry's available

(pain in the butt if you ask me). Too many expiry's, too many strikes.

(And somebody *else* gets to make out on the arbitrage.) Oh for the good

'ol days.

-Kurt

[A2] quote from this thread by Kurt

Try:

conId=0 symbol="AAPL" secType="OPT"

expiry="20110415" strike=345 right="C" multiplier=""

exchange="SMART" primaryExchange="" currency="USD" localSymbol=""

includeExpired=0

Or:

conId=0 symbol="AAPL" secType="OPT"

expiry="" strike=0 right="" multiplier=""

exchange="SMART" primaryExchange="" currency="USD"

localSymbol="AAPL 110416C00345000"

includeExpired=0

Note the two different variants on the date for expiry versus the OSI

symbol.

With futures I believe someone here said you can omit the symbol if you

specify the contract fully by localSymbol, but I've never tried it for OSI

options.

While we're at it I might as well tell you a couple other things you may run

into:

* Option chains are requested via reqContractDetails with some fields (e.g.

strike, right, and/or expiry) omitted to serve as wildcard. There is no way

to just request a list of expiries or strikes, only a full list of matching

contracts.

* You might get more reliable results in some cases (avoiding certain kinds

of errors) if you obtain a conId via reqContractDetails and use that conId

*instead of* the other fields to specify a contract for reqMarketData and

reqHistoricalData, and perhaps most importantly, placeOrder. This may be

less true now than in the past but it used to be that placeOrder was more

fussy than the other requests and would reject contracts that the other

requests accepted. Probably it is still true that reqContractDetails is

less fussy than the others due to the possibility of wildcarding, which come

to think of it (speculating) might apply even if there is only one match.

-Kurt

[A3] by dthayer

Cut and paste of code I use to get option data. If set strike=0.0 and expiry=0.0, and right="", then you get the entire chain for both puts and calls.

        Contract *C = new Contract();

        C->symbol        = "KMB";

        C->currency        = "USD";

        C->secType = "OPT";

        C->expiry = "20130621";    

        C->strike = 95.0;                        // Define as 0.0 to get entire option chain

        C->right = "P";                                // Put no string, 2 double quotes, if want both put and call

        C->exchange        = "SMART";

        C->multiplier = "";

        C->primaryExchange = "";

        C->localSymbol = "";

        C->includeExpired = 0;

        m_pClient1->reqMktData(msgID,

C,"233",false);        // false – start stream, get tick types

                                                                        // Wait in TWS::contractDetails for reply from TWS

        //m_pClient1->reqMktData( msgID,C,"",true);        // true – get snapshot, don't specify tick types.

[A4] by rholowczak

Here is a quick tutorial how to get the contract details for all options on an underlying stock (option chains) using a simple Java program and the IB Java API: Programming Interactive Brokers Java API in a Console Application – Contract Details

For implementation example using Python and Ruby see: [Q] How to get all valid options (scrape option chains)?

[Q] Why option chain data is delayed (sometimes up to 1 minute)?

[A] by IB support: (thanks to joe@higgs-tx.com)

Added on 18-Apr-2016

Unfortunately this is by design a pacing limitation to prevent clients from putting to much stress on our servers. When requesting contract details for an option chain the contractDetails () call back method will be buffered for each additional request. Below is a breakdown example on the contractDetails buffering.

For Examples:

*************************************

If Expiry = Blank or NULL

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to null or empty string, the delay is 1 minute.

*************************************

*************************************

If Expiry = YYYY (e.g. 2016)

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “2016” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year only, the delay is 1 minute.

*************************************

*************************************

If Expiry = YYYYMM (e.g. 201603)

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “201603” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year and month format (YYYYMM), the delay is 5 seconds.

*************************************

*************************************

If Expiry = YYYYMMDD (e.g. 20160318)

Symbol = “IBKR”;

secType = “OPT”;

Expiry = "20160318"

Strike = "31";

Right = "C";

Multiplier = "100";

/TradingClass = "IBKR";

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year with month plus the day (YYYYMMDD) AND Strike, AND Right, AND Multiplier or TradingClass included then there is NO delay.

Please note, that this ONLY works with front month option contracts and ONLY for the first 8 requests. If you are not requesting the front month and requesting more than 8 requests then it defaults back to a delay of 1 minute.

—- Also from jb201448@yahoo.com  —

There's also a new function reqSecDefOptParams() in the Beta version which should return option chains faster.

-Josh

[Q] Is IB’s model reliable or useless and I need to do all the calculations myself?

[A]

see this thread: https://groups.io/g/twsapi/topic/4045086#21842

[Q] Am I required to pay the dividends on shares borrowed when short selling?

[A] by Frank Bell

Dividend obligations only occur if a position is held at market close the day

before the ex-dividend date. That's true for spin-offs and all other corporate

events.

[Q] Which stocks are shortable which are not?

[A] by Jason

For a list of

shortable stocks with IB (for all countries), you can go here:

http://www.interactivebrokers.com/en/p.php?f=shortableStocks. For a complete

list of shortable US stocks, you can go here:

http://www.interactivebrokers.com/en/trading/ViewShortableStocks.php?cntry=usa&tag=United%20States&ib_entity=llc&ln=.

Lastly, for a list of US stocks that are not shortable for regulatory reasons,

you can go here:

http://www.interactivebrokers.com/en/trading/ViewShortableStocks.php?cntry=regSHO&tag=Reg%20SHO&ib_entity=llc&ln= (although

some stocks on this list may still be shortable via IB).

[Q] Unable to borrow?

[A] by Frank Bell

You can repeatedly reduce your size on unable to borrow orders until the order

is successful or you reach your minimum order size. Also, be aware that open

short orders can go from submitted to pre-submitted when you lose your borrow.

[Q] How to determine the pricing step for an option?

Hi,

Some of the options are priced with gaps, for example, if the pricing  step is 5

cents and you submit an order to buy an option for $1.02, your order will be

rejected, because the valid priced would be $1.00 or $1.05

Now, is there a way to determine the valid prices programmatically, any code or

formula (the programming language doesn't matter)

Best Regards

– JW

[A] by Kurt

Read about the minTick field of the ContractDetails structure.

-Kurt

[Q] placeOrder() No security definition has been found for the request

[A] see [Q] Errorcode(200) No security definition has been found for the request

[Q] I see duplicate events from time to time!

[A] From IB's docs:

Note:  It is possible that orderStatus() may return duplicate messages. It is essential that you filter the message accordingly.

[A] View/Reply Online (#44827)

Added on 29-Nov-2020

accepted answer

For order status you can discard duplicates by processing only messages where the remaining quantity has changed.

[A2] by Kurt

But I don't think there is anything wrong with duplicate events in this case.  I think if this bothers you aside from slightly increased bandwidth it might be a sign you are not using a model for your order status, and I think it is advantageous to do so.  You simply need to update your model from the status most recently received, and of course process all changes in status as required by your system.  It is after all very easy to detect that a change has occurred relative to the previously received and stored status.  The order status is the cumulative result of all prior activity.  As such, if a duplicate order status causes you a problem, there should be a trivial change to your algorithms that should eliminate any sensitivity to it.  It is good practice to do that anyway.  

[A3]

Also interesting that they don't send duplicate execution events, just duplicate order status events.

[Q] attached order

[A]

You should set .transmit=0 for the entry order and .transmit=1 for the

attached order.

If you set .transmit=1 for both, then TWS submits the entry order before it

knows anything about the attached order, and it may well execute before the

attached order arrives at TWS, in which case the error you described will

occur.


[Q] Bad equity options spread (bid-ask seems to widen significantly)

[A] read the whole thread


Placing orders (futures)

[Q] Can I use SMART for futures?

[A] rwk2095

SMART routing does not apply to futures.  You have to specify the exchange, and there is no extra commission for that.

[Q] How to request contract details in order to get the options for a specific future?

Gents,

Having obtained a Contract and a ContractDetails object for a specific future, I would like to request contract details to get the options chain on that future using the reqContractDetails method, but I don't know how to specify the underlying future when creating the Contract object for this request. The contract object only has a symbol field/method for the underlying, so I guess that I need to encode the future in that field, but how? Is there any field in the received Contract/ContractDetails for that future that can be used to set the symbol directly?

Contract createContractForOptionsChainRequest(ContractDetails futureContractDetails) {

    Contract contract = new Contract();

    contract.secType("FOP");

 

    // Can I use any field from futureContractDetails to set the symbol?

    contract.symbol(???);

       

    // What else needs to be set?

       

    return contract;

}

 

I'm aware that there is another call to get expiry/strike information (reqSecDefOptParams), but I am not interested in using that.

I am also aware that reqContractDetails () can be slow to return the data, but I am OK with that.

[A] by Richard L King View/Reply Online (#49441)

Added on 12-Jun-2022

If you want the full option chain returned via reqContractDetails, just give it a contract with settings like this:

Sectype="FOP" Symbol="ES" LastTradeDate="202206" Exchange="GLOBEX" Strike="0" Right="CALL"

This will return all the ES future option call contracts expiring in June 2022: there are 2300 of them this morning. Note that there are many duplicate strikes in this list because there are different trading classes (for example there are classes that expire weekly). To get just the 'standard' contracts, add:

TradingClass="ES"

to the request contract. This reduces the list to a more reasonable 280 contracts.

It's interesting to note that for some futures (for example the FTSE 100 Z futures) the options have to be specified with:

Sectype="OPT"

I've no idea why this is.

Richard

[Q] Futures – No security definition has been found

[A] by Richard L King

Added on 18-Dec-2021

ZF is on the ECBOT exchange, not GLOBEX.

 

SMART is not valid for futures: use the proper exchange name and don't set primaryExchange.

 

For local symbols in that three-part format, the first part must be four characters (followed by a space), like this:

 

ZF{space}{space}{space}MAR{space}21

 

So you want:

 

ZF   MAR 21

F1U  MAR 21

 

Similarly for the German DAX futures::

 

FDAX MAR 21

[Q] Can I place MOC order for es future?

[A] by Rob Terpilowski

MOC orders do not exist for futures on any exchanges that I know of..

[A2] by Frank Bell

MOC isn't supported for ES. What I do is place a market order with a good after

time of 16:14:58. You could probably set the time at 16:14:59, but I like to

give IB a second of grace.

Frank

[Q] Futures options data via API

[A] by souqmate (found here)

Here's a roadmap for fighting your way through IB's jungle of options on US futures (FOP).

FOP historical data are not available for expired contracts (if s.o. has recorded older data, I'd be glad to trade).

Weekly options are always European-style; they exist for forex (AUD CAD CHF EUR JPY), rates (ZT ZF ZN ZB), indices (ES NQ SPX).

Forex: have monthly American/European options: 6A/XA 6B/XB 6C/XD 6E/XT 6J/XJ 6S/XS  (actually: no XA; IB forgot?).

NG: have classes ON/LNE for monthly American/European options (but no feed nor hist data for LNE – though 4* more liquid).

ES: have weeklies (EW1 EW2 EW4), monthlies (ES in place of EW3) and EoM (EW, expire on last day of month).

NQ: have weeklies (QN1 QN2 QN4), monthlies (NQ), and EoM (QNE).

SPX (fut): have weeklies (EV1 EV2 EV4), monthlies (SP), and EoM (EV).

ZC ZS (not ZW): have usual monthlies (OZC OZS) and also monthlies whose underlying expires in Dec (OCD OSD), both American-style.

Eurodollar options (GE@Globex): GE GE0 GE2 GE3 GE4 (Globex), ED E0 E2 E3 E4 (CME).  Have no weeklies, only monthlies.  All underlying futures have (quaterly) expiries in same year (GE) or next years (GE0 GE2 GE3 GE4); all American-style.  Dunno why GE0 is not rather called GE1.  

Expiry date for GE can differ from that of GE0…GE4 for Dec month, eg 20131216 vs 20131213.

minTick can depend on expiry, eg: on 20131011, GE has minTick=0.005 for 201406 and 0.0025 for 201403.

strikeIncrement can depend on expiry, eg ZM: first expiry has increments of 5, the second of 10.

strikeIncrement can be smaller atm and bigger away: GF has strikeIncrement=0.5 close to atm and 1 away from atm; ED has strikeIncrement=0.125 close to atm and 0.25 away from atm.

SI: actual strikeIncrement is 0.05 around the money, but only 0.25 seems used.

ZC ZL ZS ZW GF LE HE (but not ZM): have factor 1/100 in strikes; only in API, not in TWS; IB won't change that.

No FOP at IB for NKD PA PL VIX.

localSymbol has no standardised syntax, eg: "6AZ3 P1075" for AUD@GLOBEX, but "C OYM  SEP 14  11600" for YM@ECBOT.

Three ways to specify contracts in queries for reqMktData or reqHistoricalData:

———————————-

symbol = GBP

secType = FOP

expiry = 20131108

strike = 1.55

right = P

tradingClass = 6B

exchange = GLOBEX

————————————-

currency = USD

conId = 132707825

exchange = GLOBEX

————————————-

localSymbol = 6BX3 P1550

secType = FOP

exchange = GLOBEX

————————————-

These work for API 9.69 or higher (since Jul 2013), where tradingClass has been pulled from ContractDetails to Contract.

For API 9.68 or earlier: only the first of these queries works; specifying conId is redundant, as strike,expiry,right are compulsory; hence for NG, forex and GE options: need to specify localSymbol when querying hist data for monthlies, to disambiguate between American/European styles (forex, NG) or midcurve options (GE0 GE2 GE3 GE4); conId is again useless here, and so are tradingClass and marketName.  

Annoying localSymbol syntax: have always 4 digits at end of localSymbol for forex  ("6SH4 P0950") or GE ("GEV3 C0112"), but 3 or 4 digits for NG ("ONX3 C900" or "ONX3 C1000"); the number is 1000*strike for forex & NG, but 1e5*strike for JPY.  For GE, have "GEV3 C9987" for strike=99.875, but "GEV3 C0112" for strike=101.125; so do printf("%04d\n",100*strike %1e4).  

However, since API 9.69, we can use tradingClass or conId instead of localSymbol to disambiguate.

Daily trade volume in IB files is ca. half of CME volume in ftp://ftp.cmegroup.com/pub/settle/stl*

I suppose the rest comes from the pit.

Happy FOP,

souqMate

[Q] How to get Futures Front Month

How do I ensure I'm always getting the active-month of futures contract?

I can either use includeExpired = True and check the date, but I would like something more robust.  Once it expires in the fourth week, I start getting this message: ERROR 1 200 No security definition has been found for the request and there's no callback to capture this.

Here's what I'm using:

        def subscribe(self):

            c = ibapi.contract.Contract()

            c.symbol = 'CL'

            c.secType = 'FUT'

            c.currency = 'USD'

            c.exchange = 'NYMEX'

            c.lastTradeDateOrContractMonth = "201706"

            reqId = self.getReqId()

            self._reqId2Contract[reqId] = c

            self.reqContractDetails(reqId, c)

I decided to use includeExpired=True and loop the month until I hit an unexpired contract

I thought there was a way to get a list of all futures expirations. Maybe I'm mistaken and that was for option chains

[A] by souqmate (found here)

Added on 15-May-2017

reqContractDetails() with

 symbol="CL" secType="FUT" exchange="NYMEX" currency="USD" includeExpired="1"

fires back 97 contracts, of which 24 expired (IB keeps 2 yrs worth of futures data – then discards them).

If you want the front month, drop includeExpired and find the youngest of the 73 contracts in the output; that won't be necessarily the most traded one.

I advise you to have your own calandar of roll dates, eg.here

[A2] by Nick

If you are going to be trading (not just academic research) almost all activity moves to the next contract before the current contract expires. This is the rollover date that souqmate advises you to be aware of and I agree with him.

[A3] by Mark

Hi Jared,

I'm new to the IB API so there's almost certainly a better way than this, but my solution was to calculate the expiry dates myself and switch to the next future as it expired.

Note that no actual harm come from requesting a wrong future, so you can suppress the error and spread your requests.

Best wishes,

Mark

[A4] by J G

If you do not specify lastTradeDateOrContractMonth you will receive information on all futures contracts.

Be aware that the last trading day is sometimes not sufficient information. For certain contracts with physical delivery you need to be out of the position before the First Position Date. See here for more details -> http://ibkb.interactivebrokers.com/node/992.

[Q] continuous contracts rollover schedule

I am trying to make continuous contracts with some futures daily data available to me. In my code, I plan to roll over to next contract based on specific dates(e.g. 1st notice day, or certain days before expiry) rather then open interest or volume. However as the rollover rules are different for different future contracts,i was wondering if anybody can help me find where to get the continuous contract rollover schedule for different futures. I have tried searching over Google and the group conversations archive but have not been successful in finding an answer. I would be much obliged if somebody can guide me.

regards,

Fatima.

[A] by souqmate (found here on old yahoo groups, now moved to here on groups.io)

below are my calendar rules for rolls (I suppose they vary from one person to the other).

They use Reuters rics, so I'll let you convert to IB symbols where needed.

When I say roll on day D, that means the new contract is used to trade on the morning of day D.

You'll find expiry dates in the output of reqContractDetails.

Gd luck

souqMate.

STXE FDX FSMI FFI IFS IFX, only HMUZ months, roll on day of expiry:

FCE MFXI OMXS30 AEX, each month, roll on day of expiry:

ES NQ YM DM TFS RMF, only HMUZ, roll on Friday prior expiry (ie 5 days earlier – Thu for YM):

CL NG, all months, roll 1 day prior expiry day:

HO RB, all months, roll 8 days prior expiry day:

GC, only GJMQZ, roll on 3rd to last day prior expiry month (was 2nd prior IBprod):

SI HG, only HKNUZ, roll on 3rd to last day prior expiry month (was 2nd prior IBprod):

PL, only FJNV, roll on 3. to last day prior expiry month:

PA, only HMUZ, roll on 2. to last day of month prior expiry month:

FGBX FGBL FGBM FGBS NK JNI JNM YAP SXF, only HMUZ, roll 1 day prior expiry day:

ZB ZN ZF ZT, only HMUZ, roll on 2nd to last day prior expiry month

FSS FEI FES JEY ED BAX YBA, only HMUZ, roll on day after expiry and always use c2; eg H7 expires 20070321, so use M7 till 20070321 and use Z7 since 20070322.

FLG CGB, only HMUZ, roll on 3rd day prior end of month preceding expiry:

YTC YTT JGB, only HMUZ, roll on day of expiry:

IND, only GJMQVZ, roll on day of expiry:

CME agric (ZC ZW ZS ZL ZM LC FC LH): roll in last week of month prior expiry month;

ZC/C: HKNUZ, skip all U contracts (except U05 20050629-20050713);

ZW/W: HKNUZ, skip most U and many K contracts;

ZS/S: FHKNQUX, use only FHKNX except Q03 and U03 (instead of X03);

ZL/BO: FHKNQUVZ, use only FHKNZ (and Q till Q04);

ZM/SM: FHKNQUVZ, use only FHKNQZ (skip Q sometimes);

LC: GJMQVZ,

FC: FHJKQUVX, switch using RDTH volumes (discarded 3 U-contracts); can skip U contracts

LH: GJKMNQVZ; use only GJMNQVZ; switch using RDTH volumes; roll on 15.-25. of month prior expiry month.

ICE futures:

CC: HKNUZ; roll in 2. week of month prior expiry month;

KC: HKNUZ; roll in 3. week of month prior expiry month;

CT: HKNVZ but use only HKNZ; roll around 15. of month prior expiry month;

SB: HKNV, roll in 3. week of month prior expiry month;

LCO/CO: all 12m FGHJKMNQUVXZ; roll 2d prior expiry  (like CL which rolls 1d prior expiry);

CME forex: URO BP JY RP SF CD AD KRW MP DX, all HMUZ (but all 12m for KRW – on Kor Fut Exch); roll  1 day prior expiry; last trade at 9:16 a.m. (Chicago time) on the 2nd business day preceding the third Wednesday of the contract month (usually Monday);

——————————————–

Souqmate,

Useful information, thanks. Do you also perform back propagation of price adjustments when you roll? To create a true continuous contract historic data file one would need to adjust prior prices at each contract roll no? Rather like back adjusting for splits for equities.

————————————————-

Yes, I roll by difference, using the difference of the settement prices (or last trade prices prior settlement time on eve of roll date) between the old and new contracts. That gives me a continues series for computing PnL in strategies.

souqMate.

[Q] Continuous contract – how to determine offset?

[A] https://groups.io/g/twsapi/topic/4044951

[Q] How to create a true continuous contract historic data (futures)? Do you also perform back propagation of price adjustments when you roll?

Souqmate,

Useful information, thanks. Do you also perform back propagation of price adjustments when you roll? To create a true continuous contract historic data file one would need to adjust prior prices at each contract roll no? Rather like back adjusting for splits for equities.

[A] by souqMate found here

Added on 19-Aug-2016

Yes, I roll by difference, using the difference of the settement prices (or last trade prices prior settlement time on eve of roll date) between the old and new contracts. That gives me a continues series for computing PnL in strategies.

souqMate.

[Q] Futures: what is “First Position Date”?

[A] by Malcolm Haylock

Added on 20-Aug-2016

Thanks for the roll schedule. Having to maintain this is obviously sensitive to changes in the contracts.

The first position date is part of the contract. Open the contract details from the IB website and you’ll see it:

https://pennies.interactivebrokers.com/cstools/contract_info/v3.9/index.php

For many contracts the liquidation date depends on the first position date:

http://ibkb.interactivebrokers.com/node/992

If there was a way to retrieve through the API the first position day then it would be much simpler to create a rollover strategy.


Combo Orders

[Q] Placing Combo orders?

[A1] (about troubles with placing combo order using paper account)

Found here: View/Reply Online (#197)

Added on 29-Aug-2017

In case somebody runs into the same issue, I talked to IB support.  They escalated me to the Trade Support Team who said that this is a known problem with combo orders in paper accounts.  Combo orders are buggy in paper accounts but they assured me that it would work fine with a regular account.  There is no known workaround to auto-transmit the combo orders other than doing it via the TWS user interface.

[A2]

has to get the "conId" thru "reqContractDetails" then add up the legs thru "addComboLeg".

todo: add an example here

[A3] by Kurt

what I do is create an object for each leg and an object for the

combo. I request contract details for each leg, with a different request

id. Then when contract details results come back the request id tell me

which leg the results are associated with. The combo object is notified

when the legs have received their contract details data. When data for all

legs has been received, then the combo object is complete, and at that point

I create the Contract object for it, with the legs, and immediately request

market data for the combo. That will both give me quotes for the combo and

also validate the combo Contract object. Or an error might come back from

the market data request telling me (error 200) "No security definition has

been found for the request", in which case I know that IB does not support

that combo or no one is providing liquidity for it (or else that I have a

bug).

One thing you may be missing is that besides the info for the legs, you must

specify the underlying symbol for one of the legs as the symbol for the

combo itself. This is a little piece of nonsense in my opinion but IB now

requires it. It is mentioned as an API topic in one of the more recent

release notes for TWS itself (not the API).

-Kurt

[A3] by fantasyfootball672

Hugh, I have a structure in here with some of the "fill"- you can easily work in your own instead

CHAR pszTemp[255];

        Order oOrder;

        Contract oContract;

        Contract::ComboLegListSPtr comboLegs;

        comboLegs.reset( new Contract::ComboLegList );

        comboLegs->reserve( 2 );

        if ( nLots == 0 )

                return -1;

        ComboLegSPtr leg1( new ComboLeg());

        ComboLegSPtr leg2( new ComboLeg());

        leg1->conId = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture1].oContract.conId;

        if ( g_Config.m_pSpreadInfos[nSpread].nContracts1 > 0 )

        {

                leg1->ratio = g_Config.m_pSpreadInfos[nSpread].nContracts1;

                leg1->action = "BUY";

        }

        else if ( g_Config.m_pSpreadInfos[nSpread].nContracts1 < 0 )

        {

                leg1->ratio = 0 – g_Config.m_pSpreadInfos[nSpread].nContracts1;

                leg1->action = "SELL";

        }

        leg1->exchange = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture1].oContract.exchange;

        leg1->openClose = 0;

        leg1->shortSaleSlot = 0;

        leg1->designatedLocation = "";

        leg2->conId = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture2].oContract.conId;

        if ( g_Config.m_pSpreadInfos[nSpread].nContracts2 > 0 )

        {

                leg2->ratio = g_Config.m_pSpreadInfos[nSpread].nContracts2;

                leg2->action = "SELL";

        }

        else if ( g_Config.m_pSpreadInfos[nSpread].nContracts2 < 0 )

        {

                leg2->ratio = 0 – g_Config.m_pSpreadInfos[nSpread].nContracts2;

                leg2->action = "BUY";

        }

        leg2->exchange = g_Config.m_pFutureInfos[g_Config.m_pSpreadInfos[nSpread].nFuture2].oContract.exchange;

        leg2->openClose = 0;

        leg2->shortSaleSlot = 0;

        leg2->designatedLocation = "";

        comboLegs->push_back( leg1 );

        comboLegs->push_back( leg2 );

        oContract.symbol = "USD";

        oContract.secType = "BAG";

        oContract.exchange = "SMART";

        oContract.currency = "USD";

        oContract.comboLegs = comboLegs;

        TagValueListSPtr smartComboRoutingParams;

        smartComboRoutingParams.reset( new TagValueList );

        smartComboRoutingParams->reserve(1);

        TagValueSPtr tagValue( new TagValue());

        tagValue->tag = "NonGuaranteed";

        tagValue->value = "1";

        smartComboRoutingParams->push_back( tagValue );

        oOrder.action = szAction;

        oOrder.totalQuantity = nLots;

        oOrder.orderType = "MKT";

        

        oOrder.smartComboRoutingParams =  smartComboRoutingParams;

        

                ORDER_ID = g_nOrderID++;

                g_pClient->placeOrder(ORDER_ID, oContract, oOrder);

[Q] Combo order example:

[A] View/Reply Online (#44252)

Added on 23-May-2020

This works fine here:

        contract = Contract()

        contract.symbol = "AAPL"

        contract.secType = "BAG"

        contract.currency = "USD"

        contract.exchange = "SMART"

 

        leg1 = ComboLeg()

        leg1.conId = 403716459

        leg1.ratio = 1

        leg1.action = "BUY"

        leg1.exchange = "SMART"

 

        leg2 = ComboLeg()

        leg2.conId = 379536992

        leg2.ratio = 1

        leg2.action = "SELL"

        leg2.exchange = "SMART"

 

        contract.comboLegs = [leg1, leg2]  

       

        order = Order()

        order.action = "BUY"

        order.orderType = "LMT"

        order.totalQuantity = 4

        order.lmtPrice = 0.01

       

        self.placeOrder(self.nextOrderId, contract, order)

I would check the API log.

Also "ratio" field MUST be integer, not float:

Yup, that is the tip.

I did see a mention of a similar story where the _actual_ issue was obscured because the NumberFormatException was happening when the Java code was trying to format a prior exception.

Useful info from the API log is shown below. It shows the NumberFormatException for "SMART", but also shows another (as a traditionally formatted Java exception & call stack) for the value "1.0".

So it turns out the real issue is that I had specified floats for the "ratio" field in each ComboLeg.

Looks like ints need to be supplied for those, because once I did, it works fine.

Thanks for the pointer, Josh.

Cheers,

Tim

[Q] Get info on all legs of placed combo order?

The use case is that I want to adjust the exit price for the order based on the percent of max gain/loss achieved and the days til expiration.  To do that, I need to get from the combo to the individual legs.

[A] by Josh in this thread

Added on 17-Oct-2016

(quote)

Partially answering my own question….

I'm now able to walk the ComboLegList in the Contract associated with the Order and retrieve the contract ID (conId) associated with each leg.  If I could retrieve the Contract based on the conId, I would be all set.  But I cannot find a get() or other method to retrieve the Contract.

Any ideas?

— Bob

——————————————————————————————-

Bob- You can call reqContractDetails with the conId/exchange (or even just the conId) to receive all the contract details.

Josh

——————————————————————————————-

Josh,

Thanks for the reqContractDetails tip.  For anyone else interested, here are a few more details on parsing combo orders.

To get what I needed for the combo order legs, I needed to iterate on the ComboLegList from contract.comboLegs in the openOrder callback to get the Ratio and Action for the leg trade, and then process the reqContractDetails call as suggested to get the Contract object for the leg, which in turn leads to the Symbol, Right, Strike, and LastTradeDateOrContractMonth..

— Bob

(end of quote)

[Q] Note: good-after-time (GAT) orders are not supported for ("generic") combo orders!

[A] by chuckfarley974@yahoo.com

Just a heads-up that took me a while to sort through this morning: Via API or not, good-after-time (GAT) orders are not supported for ("generic") combo orders, despite what the online documentation would have you believe.

[Q] Can place orders for each leg, but can’t place combo!? wtf?

[A1] by Kurt from this thread

If you can submit a request for the individual legs but can not submit a

request for the combo, then it may indicate that IB does not permit trading

on that combo. Others have noted that stock/stock combos are not available

for all pairs of stocks either. Testing, I see that future spread combos

are permitted for ES, but not for CAD december-september.

A quick call to IB will determine whether GLOBEX currency future combos are

supported.

-Kurt

[A2] by Kurt from this thread

To make a long story short, it looks like you did not specify the symbol

"CL" for the combo. I see the documentation is incorrect about this (saying

you should specify "USD"), and the API got more fussy about it a year or so

ago. Maybe that's your problem. And otherwise try "SMART" as the exchange

for the *combo* only since that is what worked for me.

Read on for the gorey details, which I already wrote before I figured that

out.

As a first step I would request market data for the combo, or historical

data. It is possible this might work even though placeOrder does not. If

so it might narrow it down to a specific fussiness about individual fields.

placeOrder has historically differed slightly from reqMarketData in the

Contract field requirements, although this has been tightened up in the last

year or so.

The only thing I wonder about is whether the exchange for the combo itself

should really be "NYMEX" or not. But I don't do futures. It makes sense in

the case of "SMART" option legs that the option combo be "SMART". But I

could also route an option combo to "ISE" even though I'm not sure "ISE"

would be correct for the legs. Just a thought.

I just tried entering this into my UI which handles some of the details:

CL:CLJ1@NYMEX-CL:CLH1@NYMEX

and I see that I can get market data (paper and real accounts) and also

placed an order in the paper account and got status PreSubmitted with no

errors in the log. However I see that my code filled in "SMART" as the

exchange for the combo even though NYMEX was used for the legs. And it

worked, so maybe try that, unless you really needed a directed order (which

costs you). The combo also worked regardless of whether I defined the far

or near leg as the long ("BUY") leg in the combo.

From my log, these are the Contract fields used for the legs:

14:42:47 sent MarketDataRequest 650 conId=37955043 symbol="CL" secType="FUT"

expiry="20110222" strike=0 right="" multiplier="1000"

exchange="NYMEX" primaryExchange="" currency="USD" localSymbol="CLH1"

includeExpired=0 secIdType="" secId="" comboLegsDescrip=""

14:42:47 sent MarketDataRequest 651 conId=37955046 symbol="CL" secType="FUT"

expiry="20110322" strike=0 right="" multiplier="1000"

exchange="NYMEX" primaryExchange="" currency="USD" localSymbol="CLJ1"

includeExpired=0 secIdType="" secId="" comboLegsDescrip=""

and for the BAG:

14:42:47 sent MarketDataRequest 652 conId=0 symbol="CL" secType="BAG"

expiry="" strike=0 right="" multiplier=""

exchange="SMART" primaryExchange="" currency="USD" localSymbol=""

includeExpired=0 secIdType="" secId="" comboLegsDescrip=""

And it looks like you did not specify the symbol "CL" for the combo. Maybe

that's your problem.

And otherwise if there is any needed info omitted from the above, let me

know and I can look at fields in the debugger, or add info to my logging

since my intent is to log everything that might be useful.

-Kurt

[Q] Is there a way to get contract details of the BAG contract itself?

Is there a way to get contractdetails of the BAG contract itself. I get the

following error when I try to request contractdetails.

MSG= Error validating request:-'dc' : cause – 'BAG' isn't supported for contract

data request. Please ent

er a valid security type ErrCode= 321

[A] by Kurt

There is no contractDetails for a BAG. This makes sense since a BAG is not

really a contract (even though the Contract structure is used to specify

it).

You may notice that execDetails for a BAG seems to have a Contract for a

bag, but if you look at the conId it is always 28812380, the same for all

BAGs. If you think about it it would be pretty hard to have a unique conId

for each possible N-way combo. I don't suppose even a 64-bit conId could

handle the general case, although in that case probably unique conId values

could be allocated dynamically by IB according to client use.

-Kurt

[Q] Combo example please!

[A] by A X <ax88@…> taken here

private Contract createBag(int conidOne, int conidTwo) {

           ComboLeg leg1 = new ComboLeg(); // for the first leg

           ComboLeg leg2 = new ComboLeg(); // for the second leg

           Vector addAllLegs = new Vector();

 

           log.info("conid in bag "+conidOne);

           leg1.m_conId = conidOne; 

           leg1.m_ratio = 1;         

           leg1.m_action = "BUY";   

           leg1.m_exchange = "NYMEX"; 

           leg1.m_openClose = 0;     

           leg1.m_shortSaleSlot = 0; 

           leg1.m_designatedLocation = "";

         

 

           leg2.m_conId = conidTwo; 

           leg2.m_ratio = 1;         

           leg2.m_action = "SELL";     

           leg2.m_exchange = "NYMEX"; 

           leg2.m_openClose = 0;     

           leg2.m_shortSaleSlot = 0; 

           leg2.m_designatedLocation = "";

 

 

           addAllLegs.add(leg1);         

           addAllLegs.add(leg2);

 

           Contract contract = new Contract();

           contract.m_symbol = "USD";     // For combo order use ?USD? as the symbol value all the time

           contract.m_secType = "BAG";   // BAG is the security type for COMBO order

           contract.m_exchange = "NYMEX";

           contract.m_currency = "USD";

           contract.m_comboLegs = addAllLegs; //including combo order in contract object

           contract.m_symbol = "CL";

 

           return contract;

       }

[Q] IB Lag with Combination Orders

[A2] by Kurt

On 7/8/10 7:54 PM, "coolwhs" <jjsmithdds@…> wrote:

> I occasionally use the combined order feature for vertical spreads,

> butterflies, etc. with options, and I've seen where the quotes change and hit

> my bid/ask which has been sitting on their exchange, and it takes a good 1-3

> seconds for IB to realize the price has been met and execute the order,

> occasionally missing the trade altogether. Does anyone have any similar

> experiences? Is there any way to get around this rather than just resubmitting

> the order and hoping the price is still the same? Thanks.

There is an option in the TWS presets for Combos which I'll bet is what is

burning you, but this is only for two-leg spreads. I'll quote from the TWS

manual:

Add Liquidity

If checked, this feature attempts to improve the price on the combo by

providing liquidity on at least one leg of a two-legged combo by submitting

one leg of a marketable combo one tick below the market for that leg.

Leaving this feature enabled may result in the combo order not executing.

For two-legged stock-stock combos.

I've found this can work really well, when it works. I've had a lot of

combo orders go through with very good prices with this option enabled,

which I think was checked by default, without my knowing of its existence.

However now and then I get burned and could not figure out why execution did

not happen when my offer was better than the best after some price movement.

So I called IB about the failed executions and they pointed me to that piece

of doc.

My impression from experience so far is that this feature works well for a

lot of things, including AAPL options, and it definitely does not work well

at all for options on SPDRs, etc. (maybe because they have a dense array of

strike prices?). I can't really draw solid conclusions yet but that is my

impression.

I don't know whether there is a way to express this option via the API, or

whether you just have to set it in TWS–never looked.

Regarding combos with 4 legs for example, I often try to decompose them into

two separate two-leg combos each of which has a relatively small net delta,

so that I do not mind submitting the orders separately and taking the risk

that they execute at somewhat different times. I've just had too many

executions fail with 4-leg combos, so I've just kind of given it up. You

might want to check whether the combo type you are interested in is native

to the exchange (maybe ISE). An IB rep told me once that combos beyond 2

legs may not be native. I never really checked whether this was true, but

just made my own adjustments to get more reliable execution.

I've also had particular executions problems with stock/option combos with

more than one option leg, and tend to decompose these as well.

Mind you the above is all based on a limited amount of *manual* trading

experience. People who do ATS stuff are more likely to have solid data

regarding these considerations. But OTOH it is not legal to do clickless

trading on US options.

-Kurt

[Q] Can’t change qty on combo order!

[A]

  • Eric J. Holtman
  • Message 1 of 7 , Jul 21, 2010
  • View Source
  • It's a bag to buy 10 strangles on a US Equities (so, one call, one put).
  • when I try, I get the following error:
  • error (5, 105, 'Order being modified does not match original order')
  • steps:
  • I place the order. It shows up correctly in TWS as the correct
  • combo, legs are correct, ratios, etc.
  • I later change *only* the totalQuantity field in the Order
  • struct, I do not touch the Contract struct at all, and I call
  • placeOrder again, and that error pops up.
  • So I know the error isn't of the type that can occur if I try
  • to use data returned back to me in a status message. It's
  • the exact same structures (except for that totalQuantity) field
  • each time.
  • Any clues?
  • Reply
  • Kurt Bigler
  • Jul 21, 2010
  • View Source
  • Do I recall someone saying that that for some kinds of orders modifying the
  • quantity is not permitted? If so this would be a case of a gibberish choice
  • of error message which we all know is typical of IB.
  • There was a thread maybe 9 to 12 months back about what fields can and can
  • not be modified in various scenarios. Maybe the native combo markets do not
  • like quantities to change or have unusual fees for such changes? I can't
  • imagine why though.
  • -Kurt
  • Show message history
  • Reply
  • Eric J. Holtman
  • Jul 21, 2010
  • View Source
  • On 7/21/2010 7:43 PM, Kurt Bigler wrote:
  • >
  • > Do I recall someone saying that that for some kinds of orders
  • > modifying the
  • > quantity is not permitted? If so this would be a case of a gibberish
  • > choice
  • > of error message which we all know is typical of IB.
  • >
  • >
  • Just tried changing either qty only or price only (I was doing
  • both at once). Neither one works.
  • It lets me do it inside TWS (of course, sigh).
  • [Non-text portions of this message have been removed]
  • Reply
  • Jason Sapp
  • Jul 21, 2010
  • View Source
  • Kurt,
  • Perhaps you are remembering the problems I had with modifying the quantity of
  • child orders of a parent order that was partially filled.
  • Jason
  • ________________________________
  • From: Eric J. Holtman <eric@…>
  • To: TWSAPI@yahoogroups.com
  • Sent: Wed, July 21, 2010 9:02:03 PM
  • Subject: Re: [TWS API] Modifying a bag order returns err 105
  • Show message history
  • Reply
  • Eric J. Holtman
  • Jul 26, 2010
  • View Source
  • After an Email from Ray @ IB, this has been solved.
  • I wasn't filling in the exchange field on the comboLegs.
  • Apparently, you can place orders without filling that in
  • but you can't modify them.
  • Once I changed it to put "SMART" in the legs,
  • everything works.
  • Show message history
  • Reply
  • Eric J. Holtman
  • Aug 13, 2010
  • View Source
  • On 7/26/2010 7:36 PM, Eric J. Holtman wrote:
  • >
  • > After an Email from Ray @ IB, this has been solved.
  • >
  • > I wasn't filling in the exchange field on the comboLegs.
  • >
  • > Apparently, you can place orders without filling that in
  • > but you can't modify them.
  • >
  • > Once I changed it to put "SMART" in the legs,
  • > everything works.
  • >
  • And of course, today, for no reason, it's stopped working
  • again.
  • I love TWS!
  • [Non-text portions of this message have been removed]
  • Reply
  • Eric J. Holtman
  • Message 7 of 7 , Aug 13, 2010
  • View Source
  • On 8/13/2010 1:35 PM, Eric J. Holtman wrote:
  • >
  • > And of course, today, for no reason, it's stopped working
  • > again.
  • >
  • Found it, replying here just in case someone searches the archives:
  • My problem was that I wasn't setting the "symbol" in the
  • main Contract (the BAG) to something sane…..
  • that is, I was always setting it to "GLD", even if I
  • was trying to trade a spread on "SPY" options.
  • It's weird that the API will allow me to place an order
  • like that (and it is correctly specifying options on SPY,
  • even though the main bag contract says "GLD"), but
  • it won't allow to me call placeOrder again to change
  • the price.
  • Once I changed it so that the main Contract has the
  • symbol SPY, everything works.
  • [Non-text portions of this message have been removed]
  • Reply

After trade is complete

[Q] How to get all Completed Trades within the last N days?

Is there a way to receive all completed trades via IB API? It seems to only return current day trades.

[A1] by Richard L. King View/Reply Online (#38974)

If you’re using the Gateway, you’re out of luck – there’s nothing you can do to return previous days’ trades.

 

For TWS, open the Trades Log window and you can specify how far back the information will go, up to a week. Whatever is set here is what you’ll get via the API.

 

Fortunately recent versions of TWS now persist this setting across sessions: in the not-so-old days you had to remember to set this every time you wanted more than just the current day, and you had to keep the window open or it was immediately reset. IBController used to automate this, but it no longer does because it can’t drive the current UI.

(update Jun 2018: Note: First, please note IBController is now effectively superseded by IBC. You can read about the reasons for this change here. You should probably consider switching to IBC.)

 

All of this underlines a point that is often made, namely that you don’t get anything via the API that isn’t available via TWS.

 

Richard

[A2] by Scott Kister yahoogroups@skister.com via groups.io View/Reply Online (#38975) 

You can set up flex queries which can be automated and fetched with a flex query token. Those queries are very flexible and could include asking for every trade and transaction for the current year.

–Scott

[Q] tell me more about Flex Query

[A1] by Jack Jost <jqjost@gmail.com> View/Reply Online (#39683)

Added on 03-Apr-2018

Historical EOD account data can be accessed via the IBKR FLEX API in XML format: https://www.interactivebrokers.com/en/software/am/am/reports/activityflexqueries.htm

Also:

If you only need execution info – just take a look at flex reports. As far as I know data can go as far as 5 years back (reports are also limited to 5 years)!

[A2] by Sal Abbasi abbasi.sal@gmail.com via groups.io View/Reply Online (#392)

I created a flex query that outputs data in CSV format.  If you go to account management / reports  / flex queries / settings you can get a token that will last a year.  The token allows your code to  run the query by using a URL for it including the report id and the token.  I then write the CSV data to a sqlite db.  These reports can give you all kinds of detailed data about past transactions, borrow costs, etc. that you cannot get from the API.

[A3] by Scott Kister yahoogroups@skister.com via groups.io View/Reply Online (#38975) 

You can set up flex queries which can be automated and fetched with a flex query token. Those queries are very flexible and could include asking for every trade and transaction for the current year.

–Scott

Also see official docs:

"Flex Web Service Version 3"

https://www.interactivebrokers.com/en/software/am/am/reports/flex_web_service_version_3.htm

Paper vs real accounts

[Q] Accuracy of paper-trading account

[A] read whole thread

— also —

If you want random data, use the Demo account, not Paper. It gives you

garbage nearly 24/7 that relates to the previous week's trading ranges, or

something similar. (Problem is most markets aren't open 24/7 and have an

open and a close, so which strategy were you going to test against this 24/7

trading US stock? It might help for finding some bugs. But Demo account

historical data was severely broken last I checked, and so your strategies

should not depend on any reference to historical data. It might work if you

start and stop your strategy at certain times since the Demo account goes

through some kind of cycles between which historical data gets reset (to

having no history).

The paper account has real data, identical to a live account, except for

market depth which has serious problems based on what people have reported

here. But it let's people test against real data when they need that.

If you don't need real data, it would be hard to create a worse simulation

than the Demo account. I'd think anyone who could would rather write their

own simulation, e.g. of markets that actually open and close and have high

volume at the beginning and end of the day.

(That's not quite what this thread was about, so you might start a new one

for the benefit of the archives if you want to continue the topic.)

-Kurt

[Q] PaperTrading – how do they simulate fills on limit orders?

[A] is funny and informative same time: here

[Q] Paper account vs. Demo account?

[A] by Kurt (from this thread)

If you want random data, use the Demo account, not Paper. It gives you

garbage nearly 24/7 that relates to the previous week's trading ranges, or

something similar. (Problem is most markets aren't open 24/7 and have an

open and a close, so which strategy were you going to test against this 24/7

trading US stock? It might help for finding some bugs. But Demo account

historical data was severely broken last I checked, and so your strategies

should not depend on any reference to historical data. It might work if you

start and stop your strategy at certain times since the Demo account goes

through some kind of cycles between which historical data gets reset (to

having no history).

The paper account has real data, identical to a live account, except for

market depth which has serious problems based on what people have reported

here. But it let's people test against real data when they need that.

If you don't need real data, it would be hard to create a worse simulation

than the Demo account. I'd think anyone who could would rather write their

own simulation, e.g. of markets that actually open and close and have high

volume at the beginning and end of the day.

(That's not quite what this thread was about, so you might start a new one

for the benefit of the archives if you want to continue the topic.)

-Kurt

[Q] Partial fillings of MKT order on paper trading account? wtf?

[A] by poch32 (found here)

Well, I think this could perhaps solve the "mistery" :

http://www.interactivebrokers.com/en/p.php?f=tws&p1=papertrader

from that link:

————

Fills are simulated from the top of the book; no deep book access

————

and just below that one:

—————–

"The trade simulator will reject the remainder of any exchange-directed market order that partially executes. This may or may not match behavior of a real-world exchange. Market orders received while there is no quote on the opposite side will be held until the market data arrives (i.e. until the first partial fill)."

—————–

So it seems that if you happen to enter a MKT order with a size bigger than the corresponding first DOM level, the order won't go thru the next level in order to get fully filled. And after that, the remaining unfilled contracts get rejected by the simulator… well, or something like that, I'd guess…

Seems a fair explanation, but… I cannot see how a tiny, miserable 6-contract order could overflow _any_ DOM level… anytime…

[A2] (quote from here)

Limitations of the PaperTrader

Although the paper trading account simulates most aspects of a production TWS account, you may encounter some differences due to its construction as a simulator with no execution or clearing abilities. These differences include but are not limited to:

  • No support for some order types including: VWAP, Auction, RFQ, and Pegged to Market.
  • Fills are simulated from the top of the book; no deep book access.
  • Limited combo and EFP trading.
  • Stops and other complex order types are always simulated in paper trading; this may result in slightly different behavior from a TWS production account.
  • Penny trading for US Options is not supported. You will be able to submit the order but it will not receive a penny fill.
  • The trade simulator will reject the remainder of any exchange-directed market order that partially executes. This may or may not match behavior of a real-world exchange. Market orders received while there is no quote on the opposite side will be held until the market data arrives (i.e. until the first partial fill).
  • Mutual Fund trading is not supported via IB PaperTrader.

(end of quote)

[Q] I am wondering if there is any way to tell from the API whether the socket is connected to a paper trading account or a real account?

[A] by Vlad Palnik

Paper account begin with a DU, execDetails.execution

[A2] by Castedo (added on 28-Apr-2015)

Dmitry's FAQ currently lists accounts beginning with "DU" as paper trading accounts, which is correct for non-advisor paper accounts. In addition, "FINANCIAL ADVISOR" paper trading accounts start with "DF". This is analogous to the "edemo" vs "fdemo" demo IB logins.

Both types, "DU" and "DF" paper accounts can be used for free with Brokertron Gateway for IB.

-Castedo (developer of Brokertron Gateway for IB)

[Q] Trying to place pegged to stock orders through API or TWS via paper account?

[A] by nirias999 (src)

If you are trying this against a paper account, it isn't working because the pegged orders aren't supported with paper accounts.


Charting

[Q] Any easy way to do charting?

[A] by Leno (found here)

Hello Vitali

I use Ninja Trader for my charting. It's free and the charts look great. Only problem, for me, is that the candlesticks times are at the end of the period, rather than the beginning, like with TWS and ESignal charts.

I use my own API for analysis and placing orders.

I haven't tried using Ninja for placing orders because I believe you have to subscribe in order to do that, but I'm not sure.

Good luck,

Leno

[A2] by Dmitry (27-Apr-2015)

Another option to draw timeseries using JavaScript in your browser is:

http://www.highcharts.com/

Here’s an example screenshot of 1sec bars view of historical data. When you zoom out highchart will rebuild bars size to show certain number of bars on the screen (you can see 5sec bar on the screenshot even though highcharts have 1sec bars data), which is configurable:

very similar result can be achieved by using Google Charts, but when you select time range with your mouse on the graph it would zoom into selected range instead of just scrolling timeline right-left. Also I’d rather go with highcharts since I found large community behind it and it was easier to overlap my datapoints (like orders or other custom indicator lines) on top of charts and it is generally working great for me (and I’m a bit lazy to study >1 framework for the same purpose:).

Apart from Highchaarts and Google Charts you can find lots of other charting libraries. Please visit collaborative comparison table about JavaScript Graph and Charts library for data visualization here:

http://socialcompare.com/en/comparison/javascript-graphs-and-charts-libraries


Fundamental data

[Q] How to get SPY dividends?

Hi,

I try and retrieve dividends from IB from tick type 59 on a few symbols here.

The returned string can easily be parsed into fields however I seem to get the wrong DY value for SPY

Other symbols such as IWM or AAPL look fine

Dividend String: 7.15705,6.284,20200918,1.578

Dividend for SPY=6.28%

Dividend String: 2.02671,1.85,20200923,0.346

Dividend for IWM=1.85%

Dividend String:  3.13,3.31,20200807,0.82

Dividend for AAPL=3.31%

Any reason for this ?

Thanks in advance

[A] View/Reply Online (#44755)

Bruno,

6.284 is the sum of dividends for the next 12 months, not including JUN 19'20 (1.40)

SEP 18'20 1.578

DEC 18'20 1.753

MAR 19'21 1.376

JUN 18'21 1.577

Total: 6.284

Just for your information: you can also obtain dividend data through the fundamentals API. I think it's the "ReportsFinSummary" report that sometimes contains two more versions of the dividend data.

My experience is that these 3 sources can contradict each other. Sometimes information is missing from one but not from the other. So it's worth comparing them.

In my experience, the dividend information can be pretty unreliable, so be careful. It will probably be ok for the big stocks and ETFs.

I have also inquired with Interactive Brokers about the "dividend schedule" that you can see in TWS (right click on symbol), but this is apparently not exposed to the API.

Best regards,

Peter

[Q] reqFundamentalData() – How to Parse XML File?

[A] by souqmate from here

Little update on Reuters' Fundamental reports:

Since API 9.68 (I think), we can use new names for snapshot/finstat/estimates, viz ReportSnapshot/ReportsFinStatements/RESC.

We also have three new types, ReportsFinSummary, ReportRatios, CalendarReport, of which only the first has currently data.  In ReportsFinSummary we find 3 reportTypes (TTM/A/R/P – twelveTrailingMonths/audited/restated/preliminary?). Eg for MSFT: these 3 types differ for the EPS of 2013-06-30, but they don't differ for the Revenues.

The parsers from xml to csv for ReportSnapshot/ReportsFinStatements/RESC were offered on 20130708; here's the parser for the new ReportsFinSummary file (much easier!):

cat $x.ReportsFinSummary.xml | awk -F\" 'BEGIN{print "# what,curr,asofDate,reportType,period,amount (but 8 fields for Dividends: what,type,exDate,recordDate,payDate,declareDate,amount)"} ! /Summary/{if(/currency/) curr=$2; if(NF>3){split($0,a," "); sub("<","",a[1]); printf a[1]","curr","; for(k=1;k<=NF;k++) if(k%2==0) printf $k","  ; gsub(/<[^>]*>/,"",$0); sub(/  *|\.000000$/,"",$0); print $0}}' >! $x.ReportsFinSummary.csv

Eg:

DividendPerShare,USD,2014-09-30,TTM,12M,1.120000

EPS,USD,2014-09-30,TTM,12M,2.582009

Dividend,USD,CD,2014-11-18,2014-11-20,2014-12-11,2014-09-16,0.310000

# to print the Dividends (8 fields) into separate file: cat $x.ReportsFinSummary.csv | awk -F, 'NF==8' >! $x.ReportsFinSummary_divi.csv

Good luck,

souqMate.

[Q] How often Fundamental data updated?

[A] by Angel from IB support

Fundamental Data is updated directly by Thomson Reuters, IB delivers the raw XML data and does not manipulate/parse it for the API. Thomson Reuters updates the Fundamental Data daily. 

[Q] Why only 4 out of 6 fundamental requests succeed and for 2 of 6 I got errors:

– failed to fetch

– not allowed

[A] by Angel from IB support

"Falied to fetch" Meaning it is not being received by Thomson Reuters, more than likely they are not providing this report type. 

      And "not allowed" seems to be a bug (they're fixing it right now and will follow-up:) [as of Aug’2014]

[Q] You mentioned "we no longer directly provide support for API Ver. 9.67". How does this work? Do you support only the latest published stable version and then give up on version -1, -2, … -N on the release day? What are the rules here?

[A] by Angel from IB support

We publicly provide the current version, one previous build and (if available) a beta version. If there are any issues with the production version of the API Components we will roll out an update to that specific version to resolve said issue. For older builds you can still use it to connect to IB but we remove them from our public website and no longer provide support for fixing any issues.


IB Data Subscription

[Q] How IB throttles / samples it's level 1 data? (aka "best description of

what's going on with the IB feed")

[A] by Richard L King: View/Reply Online (#43104) and "fudge" factor for tickSize event

Added on 19-Dec-2021

Src: Aug-2007 topic: "fudge" factor for tickSize event

Actually IB never accumulate anything (except total volume) and they don't

throttle. They just don't report every trade or every bid and ask.

You may find the following of interest. It is a composite of several posts

I've made here in recent months, and is based on some very intensive

research into how the IB data feed works that I undertook some years ago (as

far as I can see, it still works the same way today).

The IB datafeed is optimised to ensure that it keeps up with the market no

matter how busy the market is. As a result of this IB doesn't report every

trade, or every bid or ask, for any instrument in any exchange.

To accomplish this, it effectively sends a price snapshot for each

instrument at regular intervals. This interval seems to be about 300

milliseconds. For each of bid, ask, and last it compares the current price

and size with the values at the last snapshot. If the price is different it

sends both price and size. If the price is the same, but the size is

different it sends only the size. If both price and size are the same, it

doesn't send either. If there have been any trades since the last snapshot,

it sends the (accumulated) volume (so where the price and size haven't

changed but there have been one or more trades, this can be detected from

the increased volume).

By doing this, IB knows the maximum bandwidth required for each ticker, and

hence for each customer (since the number of tickers is limited), and so it

can size its servers to be able to cope with that load. If a market becomes

very busy, it makes no difference because it will still only send an update

three times a second or thereabouts, even if there have been 100 trades

during that second. This avoids the problem that every other data feed seems

to have, where the data will sometimes lag way behind the market at busy

times (with every other vendor I've used, I've had occasions where the data

could be anything up to two or three minutes behind the market).

There is an irritating side effect of this technique, which is that price

movements between shapshots may not be reported at all: for example if the

last price at snapshot 1 is 100, and then price moves up to 102 and then

back to 101 by snapshot 2, the price reported at snapshot 2 will be 101, and

the 102 price will not be reported at all. This leads to occasional

incorrect highs and lows of bars, but rarely by more than one tick: whether

that is significant depends very much on the trading strategy used.

In my experience, the number of occasions where the inaccuracies have any

significant effect is so small that it can be ignored, but that of course is

partly a function of the particular strategy I was using – note however that

it traded aggresively off 1-minute bars and used very tight stops, so you

might think it would be quite sensitive to inaccuracies. Besides, sometimes

the inaccuracies work against you and sometimes they help you, so overall I

think it pretty much balances out.

The above isn't a complete description, but it covers the basic mechanism. A

word of caution though: this is not an exact science. It would be nice if

what I said was an exact description of how it works, but you'll find odd

things happening occasionally, such as a volume update without a prior size

message where the increase in volume is not an exact multiple of the most

recent size message, or multiple last price/size messages sent at the same

time, or volume messages with a smaller volume than the previous one! But

most of the time my description is accurate.

Another point worth making is that the snapshots are taken at different

times for different TWS sessions. This is sensible from IB's point of view

because it means they don't have to send updates to all their thousands of

users at once. But it has the interesting side effect that if you run two

copies of TWS (eg different logins to the same account) you'll get slightly

different data from them, because the snapshotting is not being done

simultaneously.

By the way, one gotcha that I didn't mention above is that when both price

and size messages are sent (in a single TICK_PRICE socket message), TWS also

sends the size again in a separate TICK_SIZE message, but the volume is

correctly updated only once. I think the reason for this duplication is that

before the version 2 TICK_PRICE message was introduced, it didn't contain a

size field, so prices and sizes were always sent separately: if TWS didn't

send the duplicate size, then programs that relied on the separate TICK_SIZE

message would no longer work properly unless they were amended and

recompiled.

For backtesting, I've always used the IB realtime data which I collect in an

SQL database, and I've been perfectly happy with that. The fact is that

backtesting is very much only an approximation, especially if your strategy

uses any kind of limit orders: with market or stop orders you know that your

order would have been filled, but you can't be sure of what price you would

have got, whereas with limit orders you know at least the worst price you

would have got but you can't be sure you would have been filled at all

(unless price actually goes beyond your limit) – and your strategy might

behave quite differently subsequently depending on whether an order was

filled or not.

As an alternative to using the collected data for backtesting, one could

instead use the historical data supplied by IB – I believe that it reflects

the complete information sent by the exchange. The problem here is that IB's

historical data is not always complete, even for periods where the realtime

data feed was ok. One could also try a hybrid approach where you collect the

realtime data and then update it later on with whatever historical data you

can get from IB. But in my view, this is overkill.

Richard

[Q] Is HistoricalData volume represented in the hundreds?

[A] by Josh View/Reply Online (#44876)

Added on 08-Jul-2018

The multiplier is specified by the mdSizeMultiplier field of the ContractDetails class. For most US stocks it's 100.

The historical data from IB is intended to represent trading occurring at the NBBO so it is generally filtered for trade types that can occur away from the NBBO such as odd lots. (mentioned here)

[Q] NYSE auction order imbalance data?

I just subscribed NYSE auction order imbalance data and found out the imbalance (field – 36) is total imbalance…

anybody knows how to receive market imbalance by field-36 ?

[A] View/Reply Online (#44333)

Added on 27-May-2018

It's available with the real time market data feed- you just specify generic tick type 225 in the generic tick list parameter of EClient.reqMktData, and then the auction data will be passed back during the opening and closing auctions in tick types 34, 35, and 36.

[Q] How to download broad market statistics, such as NYSE  TRIN, NYSE Advancing issues, NASDAQ ?

[A] by  Mario Pisa mario.pisa[at]gmail.com View/Reply Online (#40162)

Added on 04-Jul-2018

https://www.interactivebrokers.com/en/software/tws/usersguidebook/mosaic/marketStatistics.htm

https://www.interactivebrokers.com/en/software/tws/usersguidebook/technicalanalytics/tick___trin_market_indicators.htm

— also —

They're available from the API, e.g.

 

Symbol: TICK-NASD

Security Type: IND

Exchange: NASDAQ

Currency: USD

[Q] tickPrice vs tickString?

[A] read this interesting thread (about 38 messages). 21-Nov-2017 update: after groop moved to https://groups.io/g/twsapi the link to original discussion is now broken : (    todo: try to find the discussion and update the link…

[Q] Would I get the same data if I subscribed to the same instrument from a paper and real account? Would it count as "one line"?

[A] by Richard L. King: View/Reply Online (#46527) 

Added on 16-Feb-2021

When I first starting investigating this mechanism back in 2003, every market data line in TWS and every market data request via the API counted against your allocation of 100 tickers, even if you requested the same ticker more than once. This was presumably because the request was allocated to a different server, and in fact the data streams received were different.. But at some stage in the late noughties (I think) this was refined so that the same request from the same user was only counted once: so now, if I request data for a given instrument from several different API client programs, that is only counted as one market data line, and they all receive exactly the same data (and that even applies whether the data is requested via the live or the paper-trading account).

[Q] Can I get historical price data of instruments which have expired?

[A] by J G View/Reply Online (#38987)

You can get historical price data of all instruments which have expired up to two years ago. I don't know how long price history currency options have. Let's assume six months. Then the oldest price you can get is if you take the option which expired 2 years ago and ask its entire price history. The oldest price point would then be 2.5 years ago.

Does this answer your question?

[Q] Trust daily close or 1-minute data close?

I am comparing the Daily (stocks) and the 1-minute data I pulled from IB, and there are quite a few differences.

Mostly it's the Close that differs, and on average by about 2%. This seems quite large to me. When I filter out the low-liquidity stocks, there are still differences, though a lot fewer.

From what I've read here, the difference may occur when a number of orders come right at the close and some go a few milliseconds past the closing time, which get bunched up to the next minute but will end up being counted as daily close.

Given the potential difference of 2%, what do you trust for backtesting?

View/Reply Online (#42498)

[A] by Julian Sommerfeldt julian.sommerfeldt [at ] gmail dot com View/Reply Online (#42516)

Added on 31-May-2019

There are some things to consider for the difference between those close prices: Auction/Time/Exchange

1) There is a close price coming from the auction at the primary exchange. This auction can take place depending on the exchange anytime after including 16:00:00, but also even some minutes later. I would say in most cases in the financial world this is the used close price.

2) There is a price coming from minute data and the last minute is 15:59 which ends at 15:59:59 not 16:00:00. So the normal close (and auction) price isnt even included in the last minute.

3) Be aware there can be a big difference if you request data directed to an exchange or as SMART (including all available exchanges).

You can get a good understanding by looking into the "Time & Sales" window showing each trade with its exchanges and conditions, e.g. auction and additional information why a trade is not reported as last price. Additional take a look at a minute chart and set "Show data outside of regular trading hours" to true, e.g. MSFT 2019-05-30 16:00 has a very much higher volume than 15:59 because it contains the auction and this minute bar is normally not shown and not included in the regular trading hours minute data.

So if you want to know the close price of the primary exchange you can do a directed request for dailys. But if you are routing SMART and not e.g. MOC orders (participating at the auction), also the data around the closing can be interesting. At the end it is a question of your trading approach.

Have fun,

Julian

[Q] VWAP through the API

[A] by MattScarpino mattscar@gmail.com

View/Reply Online (#42734) 

Added on 18-Jul-2019

If you're looking for real-time VWAP values, you can call reqRealTimeBars and set whatToShow to TRADES. You can also call reqMktData and add 233 to the genericTickList.

[Q] What RTVolume fields are?

[A] by rwk (from this thread)

When you specify generic ticktype "233" in the third parameter to reqMktDataEx() to request RTVolume the data is returned in a callback to tickString() in a single string containing six semicolon-separated data items like this:

1239.25;50;1312388185479;2439001;1244.86324288;false

According to Raymund in the API group, RTVolume is "realtime volume", and the six data items returned are last trade price, last trade size, last trade time (EDITOR UPDATE: this “last trade time” is actually generated on the TWS/Gateway side (client side) and is not passed by IB via network.. It is completely fake and bogus see this discussion for more details), total volume, vwap, and single trade flag. This is good stuff, and just what I have been looking for.

By way of background, the analysis of an order book, also know as depth of market (DOM) or Level II (though the latter refers specifically to Nasdaq) actually requires two data streams. The first is the order book itself, which contains only orders (quantity bid and offered at each price), and no trades. The trades show up on the [ticker] tape, also known as time & sales. Either by itself is not enough information to build a traders' workstation.

The problem I was having was that reqMktDataEx() returns last price and quantity in separate callbacks, and mating them was extraordinarily complex. It was in sorting through the various discussions here that I found Raoul's thread. It appears that the RTVolume was created to solve just this problem. Unfortunately, it is mostly undocumented, and the semicolon-separated string format is not exactly the most user-friendly. By specifying the "mdoff" flag on the tick generic parameter, all unnecessary market data is turned off to save bandwidth. The ticker tape is returned in tickString(), and the order book (bids and offers) is returned in updateMktDepth(). This kicks butt.

Now we have "documentation". 🙂

[rwk]

—————- and also —-

Btw. the method described above is applied on api version 963, so I'm not

sure if newer versions have changed anything since then. Upgrading means I

have to retest every little callback like I did at the time.

Greetings,

Raoul

[Q] Realtime Volume timestamp sometimes goes backwards?

[A] from this thread

Added on 13-Sep-2016

Yes, it does sometimes go backwards because it does not exist. 🙂

"Realtime Volume timestamp" is not transferred by IB, but instead is generated locally by TWS/Gateway application, which in turn is adjusting it's local time knowledge about every 3-7 minutes. Local TWS/Gateway time is adjusted instantaneously with up to 1.4sec value (usually less: 100-200ms). If IB/Gateway time jumps backwards 1.0s right in between 2 RTVolume messages then 2nd will seems to be in a wrong order even though it is not.

pps: Or may be those ticks are actually sometimes delivered in a wrong order, but there's no way to tell having only IB's data subscription as IB decided not to share their time knowledge.

[Q] Is there an easy way to get the S&P 500 index component stocks?

[A] by souqmate from this thread

Added on 17-Feb-2016

there’s no easy way to do this by using API, but …

Note that you can still do it manually with the TWS:

TradingTools -> BasketTrader -> IndexPanel -> choose SPX@CBOE as Index

set Amount=10000 shares, click on Create Orders,

and 504 rows of order will appear  (don't click on Transmit !).

You can export these via File -> SaveBasket

The csv file wll contain tickers.

I doubt the API can do the same.

Gd luck

S

Editor addon: also to parse the file in bash:

resulting file has following structure:

(quote)

Action,Quantity,Symbol,SecType,Exchange,Currency,TimeInForce,OrderType,BasketTag,OrderRef,OrderCreationTime,

BUY,100,A,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,AA,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,AAL,STK,SMART/ISE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,AAP,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,200,AAPL,STK,SMART/ISE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,ABBV,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:11 EST,

BUY,100,ABC,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,ABT,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:10 EST,

BUY,100,ACN,STK,SMART/NYSE,USD,DAY,MKT,Index: SPX,Index: SPX,20160217 15:42:11 EST,

(end of quote)

# now let us parse it and extract 3rd field and skip 1st header line:

cut -d , -f 3 snp500_basket.csv | tail -n +2

A

AA

AAL

AAP

AAPL

ABBV

ABC

ABT

ACN

ADBE

ADI

* * *

[Q] Can I download the historical data of obsolete symbols?

[A] by Vishruth Vasistha from this thread

Try setting IContract's IncludeExpired to 1. You should be able to get

historical FUT data.

Thanks,

Vishruth

[Q] How to get historical data split and dividend adjusted?

[A] View/Reply Online (#45579)

Added on 08-Feb-2021

You can use "TRADES", but they are only split adjusted.

ADJUSTED_LAST is supposed to be both split and dividend adjusted. (as per their doc):

Historical Data Types

(whatToShow)

All different kinds of historical data are returned in the form of candlesticks and as such the values return represent the state of the market during the period covered by the candlestick.

Type

Open

High

Low

Close

Volume

TRADES

First traded price

Highest traded price

Lowest traded price

Last traded price

Total traded volume

MIDPOINT

Starting midpoint price

Highest midpoint price

Lowest midpoint price

Last midpoint price

N/A

ADJUSTED_LAST

Dividend-adjusted first traded price

Dividend-adjusted high trade

Dividend-adjusted low trade

Dividend-adjusted last trade

Total traded volume

 

  • TRADES data is adjusted for splits, but not dividends
  • ADJUSTED_LAST data is adjusted for splits and dividends. Requires TWS 967+.

[Q] My accumulated volume does not match the TWS one! Wtf with VOLUME?

[A]  read this discussion

(update from Feb 2021: not copy-pasting "accepted" answer here was a mistake.. Yahoo forum is no longer there, so now we can only guess what was the answer…)    🙁

[Q] Does it make sense to use separate data / broker connections for separate tasks?

Does it make sense to connect to TWS

– for data related services with a client ID, and

– for broker services with another client ID

from the same application? Is there any pro/con using this approach?

Thanks,

Zsolt

[A] by Richard L King

My opinion is that in general it would probably be of no real benefit, but

it would depend heavily on your application.

I guess you're thinking that you'd get order-related events more quickly

than if they were mixed up with the market data stream, and this is true but

the advantage may be negligible. Assuming that it doesn't take you more than

a very small number of milliseconds to process tick events, most of the time

you'd probably get no advantage at all because there are no outstanding

ticks to process. But of course if you're handling a large number of busy

market data streams, and perhaps one or more market depth streams, then this

might alter the picture considerably. I suppose it all depends on how

critical the timely receipt of order-related events is to your application.

If a delay of say 100 milliseconds (over and above the natural latency of

the exchange to IB to TWS to client connections) would be important, then it

might be worthwhile.

Because it's bound to add some complexity to your application, my advice, as

for any optimisation, would be not to bother until you've developed and

tested the application and then only do it if there is a demonstrable need.

There are a couple of other points worth adding here:

– TWS maintains separate connections to market data, historical data and

account servers (and my assumption is that order-related events go over the

account server connection, though I've never verified this by examining the

traffic). So it's only the API connection where order-related events might

get held up;

– when you make a historical data request, the whole of the data is returned

in a single API message. If your app makes large historical data requests

and/or processing of historical data is relatively time-consuming (updating

a chart perhaps), then it might conceivably be worth making these with a

separate client id.

Richard

 

[Q] API and Trading hours?

[A] by Richard L King (src)

The contract details contains trading hours information for the current and the next day’s session. I don’t use it myself, so I can’t vouch for its accuracy, but whenever I’ve looked at it, it seems pretty good.

I’m mildly surprised that you could have got as far as producing an auto-trading system without having noticed this information: it’s been in the contract details for years. Not a criticism, just an observation.

Richard

[Q] How to get the day open price for NYSE stocks?

[A] by Josh in this thread

Added on 12-Oct-2016

Sorry Nasdaq stocks are actually on Network C. Since AAPL trades natively on Nasdaq, you would need the Network C subscription, since only the Nasdaq exchange has the official opening price.

Network A would give you the official open for stocks on Nyse, and Network B would give you the official open for stocks on Arca.

Without a subscription to the native exchange, you will just receive the first traded price on a different exchange.

[Q] How to get Futures Open Interest?

[A1] by mattpearson911

Added on 24-Nov-2016

You can get daily CME/CBOT/COMEX/NYMEX open interest and other related info from the CME ftp site:

ftp://ftp.cmegroup.com/settle/

For example: ftp://ftp.cmegroup.com/settle/stlcomex

[A2] by btw12342001

Added on 24-Nov-2016

In addition to CME directly, you can get the data from Quandl which has some APIs for retrieving data.

CME NYMEX WTI Crude Oil Futures (CL) – Data from Quandl

[A3] by Isaac

Added on 24-Nov-2016

Or, try TickData.com.

Update from OP: Tickdata.com is very expensive given that I am only interested in the open-interest information (which is only available on a daily basis anyway), but CME ftp server and Quandl are excellent tips.


Data subscription: Volume

[Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] What is the sequence of tickSize() tickPrice events?

see also relevant: [Q] Building live Trades

[A] by Mike Smith (from this thread)

if you monitor the cumulative volume AND keep your own total of cumulative volume which is the sum of the "size" parameter of each "last price" sample then you have a cross-check. When your total of cumulative volume is less than IB's reported cumulative volume then samples were missed. My approach is to add that difference at the current "last price" as if a trade was made of that size. That brings my cumulative volume in sync with IB's reported cumulative volume. Doing this each time there is a cumulative volume mismatch gives me fewer trades of larger size compared to the many more trades of smaller size which actually came from the exchange. The result is all the volume is accounted for, Market Profile tables look pretty close and life is sweet – except for one thing. The API's cumulative volume does match TWS Chart's cumulative volume.

and then by btw12342001

I do the same thing except I just remember the previous cumulative volume and subtract it from the current value. It works out to the same thing except if for some reason you don't get all the Cum vol (tickType 8) messages. This happened to me in Nov. I've since re-done my code to use the tickSize (5) message if I'm not receiving 8's

and back to MikeSmith

I have spent a lot of time with both screens in front of me and I have never seen TWS update last price without a corresponding reqMarketData update. The only way I could see this happening is if TWS is streaming level 2 quotes for that symbol which means it gets more frequent updates. Might this be the case?

As for size events, I don't look for size events less than 10 ms apart or depend on timing in at all. I look for a last price event followed by a size event with no other events in between. Then if another size event arrives which is the same size and again, with no other events in between, then I ignore that size event. Then I ALWAYS look at the difference between IB's reported cumulative volume any my running total and if my total is smaller then I create a "last price/size" event out of thin air using the difference for the size value. This takes care of those rare times when I should NOT have ignored the size event, and fills in missing samples to keep the total volume in sync as well. I found this more reliable than depending on timing.

Mike

btw, in that thread Richard L King explains why IB come up with 2msg in 1 message (tickSize and tickPrice). I will copy-and-paste it here, so you don’t miss it:

[Q] How to get volume from all exchanges vs. volume from one exchange only?

[A] mentioned here by Josh: View/Reply Online (#42746)

When requesting market data for stocks, its most common to use exchange = "SMART" so you can receive the NBBO and aggregate volume from all exchanges. So you can create a Contract object with the returned conId, and then just set the exchange = "SMART", and use that to request market data.

(If you wanted to request data from Nasdaq only, then you would have to have set exchange = "ISLAND" rather than "NASDAQ" in your request).


Data subscription: Dividents

[Q] Get Dividend information?

[A] by Kurt

I can't speak to the API but tend to believe the API never functions

*better* than TWS. And TWS has proved to be pretty bad re dividends.

For a while I was playing around with EFPs, using IB's standard scanner

settings "Relevant EFPs" under the Excess Cash category. Well one day the

scanner popped up an EFP that looked really attractive to write. It turned

out that the dividend had just been announced (as nonexistent, I think or

else much lower than anticipated), but that IB's dividend info was still

showing the originally predicted dividend. This mean that the EFP's yield

valuation was way out of line with reality, looking really good, because the

future was already priced based on the knowledge that the dividend was

lower, or gone, whereas TWS was not only displaying the wrong dividend, but

computing the EFP yield incorrectly. So writing the EFP was an instant

loss, but I didn't find out until the next day.

And I believe it happened to me more than once, although it took me a while

to figure out what was going on.

In short as IB says, dividend information is provided … blah blah blah.

In other words, they have so little accountability that they are free to

screw up in major and obvious ways with no consequence to them.

So although it doesn't answer your specific question, I wouldn't trust IB

dividend info for *anything*.

(I'm guessing the API dividend info would match what shows in TWS but I

haven't checked it.)

[Q] Dividend Calendar (dec, rec, div, exdiv, amount, etc)

[A] by Josh View/Reply Online (#39066)

Added on 08-Dec-2017

The only other place for dividend information is the Financial Summary Reuters report (API function reqFundamentals). It gives historical dividend information, including for dividends which have been declared but not yet distributed.

[Q] Dividend Adjustments? Adjusted Previous Close not available on IB?

I'm trying to check for upcoming dividend payments to ensure that all indicators are correctly adjusted and to ensure that the price drop does not create a false trigger.  Although I've read posts saying that IB adjusts for dividends, this does not appear to be the case with this example so I'd love to hear from people with experience.

For example, historical IB price data for SPY (S&P 500 ETF) appears to be the raw data.  The ex-div date for SPY was 20-Dec-13 and the dividend amount was $0.98025.  

Data from IB via req_historical_data for Trades:

Date

Open

High

Low

Close

Volume

Dec 20, 2013

180.67

181.99

180.66

181.56

1,131,358

Dec 19, 2013

181.19

181.70

180.71

181.51

927,670

Data from finance.google.com:

Date

Open

High

Low

Close

Volume

Dec 20, 2013

180.68

181.99

180.57

181.56

197,086,908

Dec 19, 2013

181.19

181.70

180.71

181.49

136,298,872

Data from NASDAQ:

Date

Open

High

Low

Close

Volume

Dec 20, 2013

180.67

181.99

180.5683

181.56

196,525,900

Dec 19, 2013

181.19

181.70

180.71

181.49

136,558,000

Data from finance.yahoo.com:

Date

Open

High

Low

Close

Volume

Adj Close*

Dec 20, 2013

180.69

181.99

180.57

181.56

197,087,000

181.56

Dec 20, 2013

0.98 Dividend

Dec 19, 2013

181.18

181.70

180.71

181.49

136,531,200

180.51

It appears that only yahoo adjusts the closing price for the dividend payment (also interesting to note the volume discrepancy with IB compared with the other sources).

Does anyone know how or if IB adjusts historical prices for dividend payments and how to anticipate them?

Thank you.

 -BB

[A1] by souqmate https://groups.io/g/twsapi/topic/4046961

warning: IB adjusts stock prices for splits and stock dividends, but not for cash dividends nor mergers nor spin-offs, etc.

Perhaps subscribing to Reuters StreetEvents Calendars ($10) will provide you with future dates and amounts of cash dividends.

IB generates sometimes wrong jumps in its price series of stocks, as some (reverse) splits are made to offset spin-offs, as happened to  UNTD on 20131101; IB and google claim factor 1:7 (huge jump), while yahoo has a more realistic factor of 5.319 : 7 (due to simultaneous spin-off of FTD).

You can vote here to have corporate actions calendars accessible via the API:

https://www.interactivebrokers.com/en/index.php?f=2493&sid=9407   (fixed link)(still works Jun-2017:)

Aside, your volume discrepancy is due to an implicit factor 100 in IB's volumes on US stocks, and perhaps to your restriction to a certain exchange (instead of SMART) or to certain times;  SMART gives me a volume of 1,466,263 on Dec 20.

souqMate.

You can vote here to have corporate actions calendars accessible via the API:

[A2] by souqmate@yahoo.com

pls search the archive before posting.   See eg

https://groups.io/g/twsapi/topic/4046961

Beware that historical data may not be availabe for several days around splits dates, as it happened for BP@BVME (1:10 split on 20140310, had no data for 3 days) or for IMI@LSE (7:8 split on 20140217, had no data for 2 weeks).

Further, some factors are inserted arbitrarily though there are no splits, eg SWN@NYSE prior 20130102 has prices 100* bigger (used not to be the case, but then such things suddenly appear…).

And as a side note (no splits here), beware that some gaps appear even in liquid symbols like Canadian futures, which have no data on 20140312 (TSE60, CGB, BAX) and that IB has lost the data for good.

Thank IB to help you see glasses half-full rather than half-empty.

souqMate.

[Q] API reqFundamentals dose not return previous dividend data for index

[A] by Josh and couple others View/Reply Online (#42651)

Added on 24-Jul-2019

Unfortunately no theres not really anything similar available for index data from the TWS API.

As a workaround:

For indexes, you need to choose a proxy, such as SPY for SPX.

Or as ETF dividends which are not really any different from equity dividends from a data perspective.

[Q] Data subscription: Calendar events

Hi all,

If you go into the Event Calendar option in TWS you can see various upcoming events, e.g. we are interested in Corporate Earnings.

There do seem to be some API's that relate to calendar events but it seems you need a subscription to Wall street horizon to download the data and the page to subscribe doesn't even seem to be accessible – maybe it's only available to professional accounts?

Is this the case or has anyone found a way to access this event data programmatically? I'm thinking of looking into some sort of windows automation tool to see if I can "scrape" the data from TWS.

[A] by alpha <pmularien@gmail.com> View/Reply Online (#44591)

Added on 17-Jun-2020

Yes, you can get this data with a Wall Street Horizon subscript (available under User Settings > Research Subscriptions). I am a non-professional and I have a subscription ($5/mo).

If you're using the Python (IB-insync) API, the code to get this data looks as follows:

ib.reqFundamentalData(contract, 'CalendarReport')

This will return XML data which is pretty straightforward to parse.


[Q] I got invalid update of bid/ask which leads to CROSSED PRICE. wtf?

[A] by Kurt (from this thread)

> I noticed that IB sends updates for bid/ask in invalid order sometimes. E.g.

> bid/ask was 45.47/45.48. Then IB sends new bid 45.99 which leads to crossed

> price 45.49/45.48. The next moment IB updates ask to 45.50 and the price is

> back to normal – 45.99/45.50.

>

> How do you guys handle that rubbish price of bid 45.49 / ask 45.48?

> Thanks

I don't know that there is any general answer. There is constant potential

for invalid/stale quotes. I don't think you are getting bid and ask

literally in the "wrong order". Rather one of the two is wrong. Was the

size non-zero on both sides? You didn't mention what instrument.

I have seen maybe not crossed bid/ask, but very narrow or equal bid ask, and

basically the bid or the ask was not real, and the question is which one. I

would find out the hard way when an order against one would not execute. I

believe that in every case I saw the problem was specific to a single

exchange which had the bad quote contributing to the "SMART" quote. Once I

recognized the problem and intervened to enter quotes lines (manually, into

TWS, but you could do it automatically with reqMktData rather nicely), I

generally found that the primary exchange did not have the problem and that

some particular exchange (e.g. maybe BOX, maybe once ISE or was it ISLAND)

had the bad bid or ask.

This problem has been recognized here and alternatives discussed about how

to deal with it most effectively. People have discussed finding the

exchange that has the most volume, etc., but I think the primary exchange is

good as a general rule. Then you can get quotes from a less-potentially

polluted source than SMART. SMART can be a real liability at times as

people have commented. Basically if any exchange at all has a bad quote,

SMART will be bad, so by using SMART for quotes you maximize the potential

for problems. So if you care about this you may want to request quotes from

a specific exchange but still place your orders on SMART to get the

commission and execution advantages, depending on all the details of order

type etc.

You might search the archives for previous comments on this topic, which I

think might be 6 months or more ago. Doing a quick search in my email

folders, I see a thread "bid/ask stuck" with posts on 6/22/2009 and

6/23/2009. But I think there is something more recent and with additional

observations. If you can't find anything (yahoo archives can be a

PITA–better to have your own repository) post back and I'll take another

look. I know that I was involved in most discussions of this in the last 2

or 3 years, which might give another clue for a search.

-Kurt

[Q] Crossed quotes on IB (again)

Added on 27-Apr-2015

I noticed the market cross on TWS where the quote on my screen showed up as 11.45 x 11.44 for a few seconds…

[A] by Mike Smith (found here)

What did you specify for the exchange? If you used SMART you will get crossed markets. For BAC you should use NYSE for the exchange. SMART is designed for picking the best exchange to route orders and for market data it is not as good as specifying the exchange with the highest volume for that stock. At least that's what I have found in practice, the "why" of it is another matter.

Mike

———- also ——————

You can see what is happening if you just fill the TWS window with BAC, but choose a different exchange for each line. Also include SMART on one line. If you watch all the bid/ask prices you can see which exchange SMART got it's data from. Exactly what it is doing is unclear but grabbing prices from different exchanges like that definitely can yield weird results. One exchange may show the same bid/ask price for many minutes, on another exchange it is bouncing around like jumping bean, yet SMART will occasionally give you the very stale data from the "dead" exchange which is very much NOT a tradable price.

Mike

————— and also ————–

It means one of the exchanges is crossed. SMART includes data from all of

them. If one exchange is crossed then the "best" will be crossed.

Like Mike said create market data lines (or API requests) for each exchange

and you will see which one is crossed and is causing the problem. In most

cases the bad data comes from one of the minor exchanges or at least not

from the primary exchange.

-Kurt

[Q] Snapshot market data vs “real time” data.

I've seen people say that snapshot market data is less accurate, reliable, etc. I'd like to learn a bit more about the specifics of why it's considered inferior.

Where have you observed snapshots providing incorrect or less data whereas the streaming service was fine?

Does the snapshot sometimes not provide all fields (bid, ask, open, etc)?

Is there any discussion regarding the latency of the snapshot vs. the real time data? For example, if a ticker price gets updated to $19.11 from $19.10, how far behind the real time stream will a snapshot request be?

[A] by Rich Holowczak (a.k.a. “Prof. H.”)

Added on 16-Aug-2016

Using reqMktData, the difference between snapshot and streaming is that once a value is provided for each field (bid Price, AskPrice, bid size, ask size, last price, volume, etc.) the snapshot is done and the request is effectively canceled. I pointed out the caveat to this before that if a field has not updated in the prior 11 seconds, it will not be echoed back with the snapshot:

http://interactivebrokers.github.io/tws-api/top_data.html#md_snapshot

If you need every field, or you need to make 100% sure you are getting a value echoed back, then you should request streaming data until your fields get a value, and then cancel the request yourself.

An alternative to this is to get real time bars which would get you a set of data every 5 seconds.

Using reqMktData, the difference between snapshot and streaming is that once a value is provided for each field (bid Price, AskPrice, bid size, ask size, last price, volume, etc.) the snapshot is done and the request is effectively canceled. I pointed out the caveat to this before that if a field has not updated in the prior 11 seconds, it will not be echoed back with the snapshot:

http://interactivebrokers.github.io/tws-api/top_data.html#md_snapshot

If you need every field, or you need to make 100% sure you are getting a value echoed back, then you should request streaming data until your fields get a value, and then cancel the request yourself.

An alternative to this is to get real time bars which would get you a set of data every 5 seconds.

[Q] Market Data Stuck when requesting snapshots!

particular question was:

Hi,

I have been having trouble from time to time where the market data seems to get

"stuck" on certain values, and does not change them even though they really do

change.

This happened to me today for over an hour with the "ES".

On one system the value got stuck, meaning that I request market data every

minute (snapshot), and received the same response each minute.

On a second computer with a different account, same code, it did not happen at

this time, but also happened later on.

Also, can anyone explain the order of the responses I'm receiving.

I see Price, Price, Price, Size, Price, Price, Price, Generic, Generic.

Thanx,

Rebecca

[QUICK SUMMARY ANSWER] Don’t use snapshots.

[A1] see discussion:

https://groups.io/g/twsapi/topic/4047462#32976

[A2] by Kurt

I never used snapshots much. I played around with them once and saw really

strange things. When I reported them rwk replied:

Hide message history

On 9/17/10 6:12 AM, "rwk2095" <rwk2095@…> wrote:

> I can't help with the snapshot problem because snapshots never seemed to work

> for me.

And if you want to see the details of the problems I had messing with

snapshots, see my post

"[TWS API] No "Last" for AAPL today?" on 9/16/10

Maybe they just don't work!

===

However others seem to be using them and not having problems. Maybe

snapshots don't work well with certain instruments?

I would call in to IB about this. Really you should have called yourself

before it was too late. As has also been repeated a few times recently

there have been problems reported as being due to particular servers in a

pool. This is *somewhat* consistent with what you report although it may be

a different twist. But if an immediate problem is associated with a

particular server in the "farm" that you are connected to, you need to call

IB during market hours and before you lose your connection, else no

diagnosing can be done.

> Also, can anyone explain the order of the responses I'm receiving.

> I see Price, Price, Price, Size, Price, Price, Price, Generic, Generic.

Without the tick# values this doesn't say much, so you might as well re-post

so that people can answer without having to second-guess.

And just to be clear (which I wasn't) by tick# I meant BID, ASK, LAST, etc.

Price/Size/Generic is secondary information, and redundant once you have

specified BID/ASK/LAST etc. So if you report it here that way you may get

some information back. There has been some discussion before about the

order that ticks arrive and what it means. Some of it is determined by the

so called "sample code" (really the library), which you can read yourself,

i.e. the place from which tickPrice and tickSize are called is from source

code provided by IB which is part of your code base.

-Kurt

[Q] reqRealTimeBars vs reqMarketData – how to get real time bars?

[A] by Kurt

reqHistoricalData lets you have 3-minute bars, but only gives you backfill.

You need reqRealTimeBars. However, there is no 3-minute option. Bars come

every 5 seconds once you subscribe. You will have to coalesce them the

5-second bars into 3-minute bars, but that should be easy enough.

However I know people have posted here about having problems with real-time

bars. I can't remember the details so you might check the archives. Part

of it is that people expect the bars to match the streaming data, which it

won't. But I seem to recall other problems, like maybe volume being 0 when

it should not be.

If you have trouble with real-time bars another option is to build your own

bars from tick data (reqMarketData).

Yet another option is to call reqHistoricalData on a timer, as you were

suggesting. However your latency will probably be worse because you have to

wait until the last bar is clearly complete before you make the request.

Otherwise the final bar would be truncated. (You should probably have logic

to detect the truncated bar based on the end time on the "finished" record,

and re-issue the request if needed, and refine your timing logic to minimize

latency without getting truncation.) You *can* use reqHistoricalData to

request a single bar (in cases I have tested).

-Kurt

[Q] How to get list of all IB’s tradable contracts?

[A] by Shane Cusson

I had some free time this weekend, I fired up a crawler. It pulls down

216,000 contracts from 224 exchanges listed on the products page.

I called it "IBContractExtractor", released under GPL. It's available here:

http://sourceforge.net/projects/ibcontractextra/

Shane has moved it over to GitHub at https://github.com/shanecastle/IbContractExtractor

It's written in C#, uses the HtmlAgilityPack. The pack is an awesome

library, takes all the grudge work out of parsing HTML, gives you a full OO

interface to the contents of the Html. Right now it's just pull out the

contracts, I'll make it extract all the other details at some point in the

future. The resulting text file is about 12MB, zips down to 3MB.

I don't know if it'll run on Mac/Linux yet, I did use some .net 4.0

features. You'll need .net 4.0 for it to run.

Please kick the tires, let me know how it works out for you.

–Shane Cusson

[Q] Sometimes lastSize=0. wtf?

[A] by mikesmithv (source thread: here)

Couriosity got the better of me so I turned on debug logging and looked at my code. It turns out that what I said is correct. You will loose data if you ignore lastSize events but it's complicated. I had to refresh my memory going over code I wrote some time ago but it appears not much has changed so here are the details for what it's worth.

First you have to multiply every size by 100 for stocks for both size and cumulative volume events. It is critical that you hold the expanded numbers in your code for what I say here to work.

Next, for size events greater than 1 (or 100 multiplied) you IGNORE them if they occur immediately following a size event of the same size AND that size event was proceeded by a price event. This is the infamous "duplicate size bug" which has been there for years. If you do not ignore this your running total of size will creep ahead of cumulative volume reported by IB.

Finally, for a strange twist, if the last size is zero then you DON'T ignore it. These zero size events represent trades of less than 100 as I mentioned earlier. Since everything is divided by 100 they get rounded to zero before they are sent out. You can have several trades at different prices with zero size, so that means price was fluctuating but at a volume that was under the radar of cumulative volume so just looking at cumulative volume delta will miss this. For example if you have trades of price $100, size 10, price $101, size 20, price $102, size 30 then you would see 3 price events of zero size and cumative volume would not have changed since 10+20+30 is less than 100 which is what it would take to bump up cumulative volume by one. That's IB's cumulative volume I'm referring to, yours is times 100.

So if you see zero size for stocks (never happens in Futures BTW) then SOME amount of volume occurred at that price. The question is how much? We just know it is something less that 100. I found 20 to be a good average. It worked well for matching Market Profile totals, as least as well as much more complicated schemes I came up with. You can be as creative as you want in coming up with this figure but it is important to post SOME quantity at the last price when the size is zero. It should be at least 1 so you have some indication a stock has traded through a price at sub 100 volume.

[Q] When is the exact start of a new bar? Will hand-made bars (from tick data) be exact same bars as I receive from IB?

[A] by Kurt (src)

My memory about IB's bars is that we know that the start of a new bar

happens at a time that we, on the other end of the sampled data feed, can

not expect to identify. In fact the moment a new bar starts may not even

*exist* as a tick in the sampled data feed. So what you are getting is

basically the equivalent of "aliasing". If the price jumps up *before* the

point you think is a new bar, but IB's bar timing thinks that moment was

*after* the new bar started, then you will see the right data in the wrong

time slot. It might have been only 1/10 of a second difference in timing,

but sampling at the wrong point can make it look like it is a whole second

off. So you can expect this time skewing, and that is what your last plot

seems to show.

— also —

I don't have any information to the effect that IB's bars are based on

sampled data.

Regarding the "tick data" I've heard here that two separate connections get

different sampling. So it would seem (if it weren't already obvious) that

IB's systems have access to unsampled data. I would assume they would use

such unsampled data to generate their historical bars.

The point is that even if the historical bars come from unsampled data, they

can not be expected to match sampled "tick data" in the sense that the bars

could be reproduced from the sampled tick data.

–Kurt

— also —

It must be the case that IB take their input from unsampled data streams, surely.

Also, IB told me that it's sampled every 300ms – at least for forex, they

weren't explicit.

I assume they sample it to reduce down the ticks to 3 per second instead of the

random amount on the live feed.

What that tells me is that (a) what you mentioned about the real O and C for any

bar in the resulting dataset being fictional and (b) the H and L could be lower

or higher respectively than the actual H and L. (and of course (c) the H and the

L could appear in either bar on whichever side of a particular time point).

–Adam Hardy

[Q] Why IB uses sampled data instead of plain real-time tick streams? Aka "fudge" factor for tickSize event.

[A] by Richard L King

see:

https://groups.io/g/twsapi/topic/34510650

https://groups.io/g/twsapi/message/8821

Actually IB never accumulate anything (except total volume) and they don't

throttle. They just don't report every trade or every bid and ask.

You may find the following of interest. It is a composite of several posts

I've made here in recent months, and is based on some very intensive

research into how the IB data feed works that I undertook some years ago (as

far as I can see, it still works the same way today).

The IB data feed is optimised to ensure that it keeps up with the market no

matter how busy the market is. As a result of this IB doesn't report every

trade, or every bid or ask, for any instrument in any exchange.

To accomplish this, it effectively sends a price snapshot for each

instrument at regular intervals. This interval seems to be about 300

milliseconds. For each of bid, ask, and last it compares the current price

and size with the values at the last snapshot. If the price is different it

sends both price and size. If the price is the same, but the size is

different it sends only the size. If both price and size are the same, it

doesn't send either. If there have been any trades since the last snapshot,

it sends the (accumulated) volume (so where the price and size haven't

changed but there have been one or more trades, this can be detected from

the increased volume).

By doing this, IB knows the maximum bandwidth required for each ticker, and

hence for each customer (since the number of tickers is limited), and so it

can size its servers to be able to cope with that load. If a market becomes

very busy, it makes no difference because it will still only send an update

three times a second or thereabouts, even if there have been 100 trades

during that second. This avoids the problem that every other data feed seems

to have, where the data will sometimes lag way behind the market at busy

times (with every other vendor I've used, I've had occasions where the data

could be anything up to two or three minutes behind the market).

There is an irritating side effect of this technique, which is that price

movements between snapshots may not be reported at all: for example if the

last price at snapshot 1 is 100, and then price moves up to 102 and then

back to 101 by snapshot 2, the price reported at snapshot 2 will be 101, and

the 102 price will not be reported at all. This leads to occasional

incorrect highs and lows of bars, but rarely by more than one tick: whether

that is significant depends very much on the trading strategy used.

In my experience, the number of occasions where the inaccuracies have any

significant effect is so small that it can be ignored, but that of course is

partly a function of the particular strategy I was using – note however that

it traded aggressively off 1-minute bars and used very tight stops, so you

might think it would be quite sensitive to inaccuracies. Besides, sometimes

the inaccuracies work against you and sometimes they help you, so overall I

think it pretty much balances out.

The above isn't a complete description, but it covers the basic mechanism. A

word of caution though: this is not an exact science. It would be nice if

what I said was an exact description of how it works, but you'll find odd

things happening occasionally, such as a volume update without a prior size

message where the increase in volume is not an exact multiple of the most

recent size message, or multiple last price/size messages sent at the same

time, or volume messages with a smaller volume than the previous one! But

most of the time my description is accurate.

Another point worth making is that the snapshots are taken at different

times for different TWS sessions. This is sensible from IB's point of view

because it means they don't have to send updates to all their thousands of

users at once. But it has the interesting side effect that if you run two

copies of TWS (eg different logins to the same account) you'll get slightly

different data from them, because the snapshotting is not being done

simultaneously.

By the way, one gotcha that I didn't mention above is that when both price

and size messages are sent (in a single TICK_PRICE socket message), TWS also

sends the size again in a separate TICK_SIZE message, but the volume is

correctly updated only once. I think the reason for this duplication is that

before the version 2 TICK_PRICE message was introduced, it didn't contain a

size field, so prices and sizes were always sent separately: if TWS didn't

send the duplicate size, then programs that relied on the separate TICK_SIZE

message would no longer work properly unless they were amended and

recompiled.

For backtesting, I've always used the IB realtime data which I collect in an

SQL database, and I've been perfectly happy with that. The fact is that

backtesting is very much only an approximation, especially if your strategy

uses any kind of limit orders: with market or stop orders you know that your

order would have been filled, but you can't be sure of what price you would

have got, whereas with limit orders you know at least the worst price you

would have got but you can't be sure you would have been filled at all

(unless price actually goes beyond your limit) – and your strategy might

behave quite differently subsequently depending on whether an order was

filled or not.

As an alternative to using the collected data for backtesting, one could

instead use the historical data supplied by IB – I believe that it reflects

the complete information sent by the exchange. The problem here is that IB's

historical data is not always complete, even for periods where the realtime

data feed was ok. One could also try a hybrid approach where you collect the

realtime data and then update it later on with whatever historical data you

can get from IB. But in my view, this is overkill.

Richard

Also note: Stock data is sampled all 250ms and options data all 100ms.

[Q] What are the differences between Level I and Level II data ?

[A] found in this thread

In reference to our previous thread counting market data events, I did a

little analysis counting the number of Level 1 updates vs Level 2 updates at

the inside market.

My futures trading platform relies upon the current Bid / Ask price, which I

normally derive from the Level 1 updates.

My analysis shows, however, that Level 2 (DOM, Market Depth, the Book)

updates Position 0, which is the inside market, at about 5 times that rate.

So there might be about 200 or so updates of the Bid size in a minute, but

there might be 1000 or so updates of the inside Bid on the DOM.

On the face of it, that is more current information perhaps than the Level 1

update. Hmmmmmmmmmmm…… 🙂

So I think I need to review / upgrade my code to possibly use this more

frequent DOM update value as my most "current" value for the (derived) Level

1 Bid or Ask price. Whaddya think?

[Q] Which request ID should I use?

[A] by Kurt

The people who can answer your question are probably the people who simply

tried every possibility. I did once, but not for scanner. But as I recall

each type of request has its own space for request id's. So you probably

have a lot of freedom. But at the same time you might as well choose id's

that are more convenient for your own tracking.

For that purpose I use a unique id for each request, globally across all

request types *except* for orders, for which I use a method specific to

orders which disregards whether the same request is used for other request

types. But that's just an accident of development.

Some others use unique request id's in different ranges for different

request types, so they can tell directly from the request id what kind of

request it was associated with.

So you can do what you like. But as to your original problem I have no

idea, except I recall people have had request id uniqueness problems related

to multi-thread coding problems, critical section issues or just not

incrementing at the right point in their code. No idea whether that applies

to you.

-Kurt

[Q] How deep can I go in history?

[A] by Dmitry

Usually it is only -1 year back, but if you pay (or trade) more you can reach up to -4 years, which brings you somewhere to Jan-2010 (quite far from desired 2000:(

a quote from this page:

  • Historical data requests can go back one full calendar year or more, depending on the number of concurrent real-time market data lines:

Number of Market Data Lines

Historical Data

Request Limit

Less than 499

One year

500 – 749

Two years

750 – 999

Three years

1000

Four years

 

Market data lines can be increased based on monthly commission amounts, amount of equity and Quote Booster subscriptions.

[A2] by souqMate

That table for hist data is only for stocks are FX.

Futures contracts are removed from IB's db 2 years after they've settled.

So even if you paid enough fees, you couldn't get more than 2 years.

Options (on futures or stocks) are removed right after they've settled.

souqMate

[Q] Bug in historicals? Price but no volume and other historical “holes”!

[A] by Kurt

What is missing from historical data can be for example particular bar sizes

for particular instruments, particular values of the "what" parameter for

particular instruments, etc., like the GLD example I cited. I take it that

option implied volatility is never available for GLD.

If you work with it long enough you will find that IB's "offering" is full

of holes. Not holes in time, but holes in the consistency of the offering.

There is no doc for what you will be able to get or what is not available.

IB either provides it or they don't. In many cases there is no error

message, just an absense of data.

-Kurt

[A2] by Ron Hinchley

With stocks missing data happens when no trades take place. I download 1-second data sometimes and even some NASDAQ-100 stocks have 1-6 seconds missing at the start of the day and sometimes an hour. When I run nightly I go back into my SQL two weeks looking for holes, then re-request the missing data if anything is longer than 5 seconds in a half hour. The data seems to stay missing but I don't check. Start of trading has often has an initial gap 1-5 seconds.

–Ron

[Q] Options historical data?

[A] by Kurt

I think IB does not offer 1-day data for options. I get the same result as

you. However I get data when I request 1-hour bars. However that won't

take you back a year. It looks like I can go back about 2 months. TWS

charts behave similarly.

Welcome to IB historical data. Well maybe you're not new, can't tell for

sure from your handle. (I wish people would sign their names, especially

people who use several email addresses. It is hard enough already.)

-Kurt

[Q] How to store historical data?

[A] by Dmitry (from this thread)

Added on 25-May-2017

Alex,

You can ask for 2000 1sec bars in one request (covers 33.3minutes). Simply advancing next request by some fixed amount of time won't work because when you request get close to the market open time then IB would send you back 2000 bars where some of them from one day and some of them from previous day!!  More over, if you request was close to some weekend/holiday (say another endDateTime happen to be 9:40am on Monday) then some of 2000 bars will belong to Monday while some to Friday around 3:45pm-4:00pm.

To get continuous history for N days you basically you need to do the following 3 steps:

  1) send reqHistoricalData with endDateTime pointing to the end of the period of your interest

  2) upon getting 2000 bars (historicalDataEnd()) you'd get oldest bar:

    and figure out if you have enough (i.e. if oldest bar's timestamp is older that the start of the time range you're trying to get)

  3) if not enough, then you send another reqHistoricalData() with endDateTime argument set to your oldest bar timestamp from step #2, and you loop to step#2

The above 3 steps would create a nice and maximumly continuous set of bars with no overlaps between your requests. It would be good idea to store each complete 2000 bars in persistent storage (say DB), so if you happen to be interested in the historical data around that time it is available for reuse and you don't have to request it all over again, but there's more work to add persistent storage to your historical bars. I'll give a glimpse of potential problems:

[how you know if your stored 1-week worth of data is continuous and complete?]

Say you want to replay your backtest against 1 week of data. For simplicity sake – you have no data stored previously for a given instrument. You loop through steps 1-2-3 (with 10sec sleep between requests to avoid hitting rate limits). One RTH=true day would be 6.5*60*60 = 23400.0 bars per trading day, which correspond to 23400 / 2000.0 = 11.7 historical requests and with 10sec gap in between requests it will be about 120sec = 2 minutes to download 1 day for 1 instrument (or twice as fast if you download your history simultaneously from paper and real accounts running on the same box on different ports;)

One week of data will take roughly 5 x 2 = 10 minutes. So 10+ minutes later your loop is over and your backtest fires many times and you're happy.

Next day after your platform restart you want to continue to play with the exact same time range to test a few more ideas, but now the problem is: how does your platform know if you have a complete set for the requested 1-week period or does it have any uncovered gaps? You can not rely on the exact number of bars per day, because sometimes it will actually be less than 23.4K bars! Also sometimes there are halves of trading days with early sessions close or sometimes there are long weekends due to some holidays and the whole day might be missing, but 1-week is still continuous and there's no need to request anything!  Waiting another 10 minutes just to make sure that all 1-week stored bars are same 1-week bars available from IB does not make sense (and btw, sometimes it will be adjusted on IB end so it will be not exactly the same set of 1-week bars!-))

[how you properly merge existing "continuum(s)" with new data?]

But later on you decide you need 2 weeks of data instead of 1 week and it happens to overlap 1 week of data already stored in your (see previous step). Or in more complex cases you want a larger period (say 1 year of data), which overlaps with few existing sparse weeks inside that larder period here and there. How would you solve the challenge of "not re-requesting IB for the data you already have?". As you can see for those cases simple steps 1-2-3 might turn into something more complex.

One of the possible simple solutions would be to mark your stored bars up to "complete day" precision. Then for each instrument/contract apart from just "bars" table you'd have to add another table with the only column: "complete days" and then while you're looping 1-2-3 steps backwards in time you can always check if you oldest bar happen to accidentally hit one of your "complete days" territory and then you trim 2000 requests to avoid overlap/duplicates in DB, store that trimmed subset and then fast-forward down to the morning of the "complete day" you've just hit (or even skip several days if they go one by one in a row and they all in "complete" state).

In this solution if you have 5.5 days of history stored 5 days will be marked as "complete" and 0.5 day will be not marked, thus next time the wider history range is requested you'd discard any incomplete data and would have to re-request that day over again (until it is complete).

Another approach that I implemented with more granular visibility into "completeness" would be to have a 2nd table "continuum" with 2 columns for timestamps: "from" and "to". Each time you got 2000 bars – it is already something reliably continuous and you'd store 2000 bars along with a new record into "continuum" table with "from" and "to" pointing into the start and end of those 2000bars.  As you go with more complete historical data requests you just updating existing record for your continuum and by the end of retrieving 1-week period you have 23400 * 5 = 117000 bars saved in one table plus 1 record in your "continuum" table pointing into start time used for 1st request (can be anything, like even "midnight" btw) and end of that week (pointing to the oldest bar timestamp, because you can't use "midnight" here because you never know what data out there beyond your last requested range unless you commit to stay inside RTH=true for the rest of your life:). Then you work out the logic of merging 2 continuums as soon as they start to overlap, and that is it!  Now not a single request is waisted. Simple like that!-)

Cheers,

Dmitry

[Q] reqHistoricalData: Volume vs Count?

by Jason Jones <jasonjonesa11@hotmail.com> View/Reply Online (#42484)

Added on 27-May-2019

Can someone please explain the difference between the Volume and Count values that are returned in the Bar data after a reqHistoricalData?

IB's gives the following definitions for these properties:

Volume: The bar's traded volume

Count: The number of trades during the bar's timespan

I think of volume as "the total quantity of traded shares" which I would expect to be the same as the "number of trades".  But, the Volume and Count values are often different.

Here are some sample values returned for the Volume and Count after I requested 1 second historical data:

Volume, Count

72, 44

0, 0

2, 1

1, 1

46, 25

304, 135

20, 1

113, 51

3, 3

[A] by rwk View/Reply Online (#42485)

Back in the old days (i.e. 20th century), volume was usually not available until after the close or even the next morning.  Some traders followed "tick volume" whereby each trade caused the count to be incremented by one regardless of the number of contracts.  Some traders regarded tick volume as worthless, but others considered it [slightly] better than nothing.  Once realtime reporting of volume became available, it was often referred to as "live volume" to distinguish it from tick volume.  I doubt that tick volume gets used much anymore, especially in stocks.

(update by Nick): The count is trades, not orders. One trade (a buy and a sell) of 100 shares is  volume of 100 and count of 1.

[Q] How to backtest?

[A] by Kevin Zhu via Groups.Io <Z_kvn=yahoo.com@groups.io> View/Reply Online (#1531)

Added on 06-Jun-2018

Try Backtrader. Best pure Python backtesting open source solution I've found so far, excellent documentation.

I use insync to download data to feed into backtrader.

https://github.com/backtrader/backtrader

https://www.backtrader.com/

[Q] high, low, close fields in tickPrice ?

[A] by vishruth

My Qn to IB: In real time data access, what is the significance of high and low ? I should be just getting Bid, Ask and Last / Close only. Is that not right?

IB:

high – high price for the day

low – low price for the day

Above, they are mainly used for information. Normally, users obtain the information for trading analysis.

My Qn to IB: Also, what is the difference between last and close price returned by this API call ?

IB:

Last – The last traded price at which the contract traded

Close – Yesterday's closing price.

[Q] What is the sequence of tickSize() tickPrice events?

"If we get a size event then price event, then size event,

do we know whether the price should be associated with

the preceding size event or the subsequent price event?"

see also relevant: [Q] How to get somewhat accurate cumulative volume?

see also relevant: [Q] Building live Trades

[A0] From IB's discussion board:

We only send down changes and the order for which we check is:

Bid, Ask, Last, Bid Size, Ask Size, Last Size, High, Low, Volume, Close

So, in answer to your question the price event should be associated

with the subsequent NEXT size event (as price is sent before

size)

PRICE IS SENT BEFORE SIZE

[A1] by Dmitry:

I’d rather suggest to read IB API sources to see how and when API sends these events to your code. It’s more clear and its only about 100Kbyte+ c++ cource code to read!

[A2] blackbox_tws_api blackbox_twsapi@btinternet.com>

you can see what is going on under the hood with the EReader code.

The latest version will be >= 3 (currently version 7), so a price 'tick' sent from the IB server has the following information:

1. version of TICK_PRICE (>=3)

2. tickerId

3. tickType

4. price

5. size

6. canAutoExecute

Thus, the price tick received by the API has both price and volume information.  The API sequentially calls the tick_price and tick_size methods (note that it is impossible for the API to call tick_size again in between these two method calls, so the tick_size information immediately following tick_price is always related to that tick price).  

If the price has not changed, IB conserves bandwidth by only sending the current size information.  The size 'tick' has the following data:

1. version of TICK_SIZE

2. tickerId

3. tickType

4. size

Any size tick will thus relate to this previous price tick price of the same type (e.g. bid).

As you can see from the code below, the reader parses the tick data in lines 2-13 and then calls the tickPrice method on line 14.

After returning from that method call, it checks whether the TickType it read from line 4 was a price tick (1-Bid, 2-Ask, 4-Trade) or something else.  If it was a price tick, the EReader then calls the TICK_SIZE function on line 30.

As you can see from the code, the SAME tick read from EReade calls both the TICK_PRICE and the TICK_SIZE methods.

For example, this is the raw data for several ticks received for SPY using edemo (given that I need streaming data on a Sunday).

msgId                                    1                1                2                1                2

version                        6                6                6                6                6

tickerId                        1                1                1                1                1

tickType                        4                2                0                2                3

size                                2                32                48                30                30

price                                188.88        185.95                        185.75        

canAutoExecute        False        True                                True

msgId:          1=TICK_PRICE, 2=TICK_SIZE

TickType:        0=bid size, 1=bid price, 2=ask price, 3=ask size, 4=last price, 5=last size, 8=volume

As you can see, the data received by TWS/IBG is either a size tick or a size and price tick together.

I hope this helps to clarify what is actually happening with the tick price and tick size data.

 -BB

  1         case TICK_PRICE: {

  2               int version = readInt();

  3               int tickerId = readInt();

  4               int tickType = readInt();

  5               double price = readDouble();

  6               int size = 0;

  7               if( version >= 2) {

  8                   size = readInt();

  9               }

 10               int canAutoExecute = 0;

 11               if (version >= 3) {

 12                   canAutoExecute = readInt();

 13               }

 14               eWrapper().tickPrice( tickerId, tickType, price, canAutoExecute);

 15

 16               if( version >= 2) {

 17                   int sizeTickType = -1 ; // not a tick

 18                   switch (tickType) {

 19                       case 1: // BID

 20                           sizeTickType = 0 ; // BID_SIZE

 21                           break ;

 22                       case 2: // ASK

 23                           sizeTickType = 3 ; // ASK_SIZE

 24                           break ;

 25                       case 4: // LAST

 26                           sizeTickType = 5 ; // LAST_SIZE

 27                           break ;

 28                   }

 29                   if (sizeTickType != -1) {

 30                       eWrapper().tickSize( tickerId, sizeTickType, size);

 31                   }

 32               }

 33               break;

 34           }

 35           case TICK_SIZE: {

 36               int version = readInt();

 37               int tickerId = readInt();

 38               int tickType = readInt();

 39               int size = readInt();

 40

 41               eWrapper().tickSize( tickerId, tickType, size);

 42               break;

 43           }

[Q] I get negative and zero quotes! wtf?

[A] by Kurt

You should probably ignore quotes that are zero or negative,

unless you are trading combos. Your code should probably behave as if the

quote was not received, unless your logic depends on detecting absent

quotes, but even if so it is not a bad thing to try for diagnostic purposes.

Presumably if a negative quote causes a problem, and you ignore negative

quotes, your problem will go away. If so, then it does NOT sound like an

issue re the rate of requests. I think it is very unlikely that the rate of

your requests would *cause* IB to transmit zero or negative quotes.

Negative and zero quotes are a "valid" condition in the IB quote

stream. It is the normal state after-hours, and in some cases also when

liquidity disappears during trading hours (not unusual with US option

combos). You probably want to look at the size field as well to help

disambiguate these conditions if you are trading option combos, since in

certain cases a 0 or -1 price on a combo could be a real price. (But the

bid and ask will not *both* be -1.)

Also if you log your requests and request id's then you might well see

immediately whether you are transmitting more than request with the same id.

If you see this then you know you have a bug. Of course logging (especially

to an on-screen log window) might slow your data rate and hide a problem.

-Kurt

One correction. You can have negative bid *and* asks. I am writing some altos that scans options and very often on the way far away options there will be zero and minus one values. The minus one values annoy me to no ends as tws values them and all of the sudden I made or lost several million dollars on my currency future option positions.

Regards

Christian Gross

[Q] How to get all strike prices for a given ticker & expiry?

[A]

> First you need to invoke reqContractDetailEx(…)

> to obtain all related option products

> When invoking reqContractDetailEx(…) then you leave the one of the

> contract parameters empty, the empty attribute acts as a wildcard.

> Therefore, it will return all the valid combination products for

> that specified symbol. For example, I want to obtain all the valid

> strikes for IBM July option. In this particular case, I leave the

> strike field under contract object set to zero. Then this should

> return all the valid strikes for the month of July. See example

> parameter below:

>

> Symbol = IBM

> SecType = OPT

> Expiry = 201007

> Strike = 0

> Exchange = SMART

> Currency = USD

>

> The information delivers by ContractDetail(…) event.

>

> Raymund

> IB Technical Support

[Q] No security definition has been found.

[A] by blackbox_tws_api <blackbox_twsapi@…>

Always start with the IB Contract Information Center to ensure you have the relevant fields for your chosen contract:

http://www1.interactivebrokers.ch/contract_info/v3.8/index.php

Once you located the relevant contract, use the conid to request contract details.  You can then determine what corrections need to be made.

For your request below, there is also the potential issue of the expiration date.  In general, IB does not generally provide historical data more than one year old.

Other corrections:

Hide message history

contract.m_symbol="QQQ";

contract.m_secType="OPT";

contract.m_expiry= "20131018";                << Example using more recent contract, in YYYYMMDD format

contract.m_strike= 42;

contract.m_right = "C";

contract.m_exchange="CBOE";                << "SMART" will also probably work, depending on your requirements

Using the above and some intraday bar size settings (e.g. duration = "1 D", barSizeSetting = "5 mins"), I get an error of No market data permissions for CBOE OPT rather than the previous No security definition has been found.

Hope this is useful for you.

 -BB

[Q] reqMarketData(): For most stocks I get the last prices, however for some stocks I only get bid.

[A] by Kurt

Well that *does* happen some times. Particularly on very-low-volume stocks

I've seen the bid or ask disappear and then return a couple seconds later.

But it is pretty unusual in my experience and I don't see it in US stocks

during regular trading hours. I believe I see it during regular trading

ours on some US ETF option (native) spreads.

However, I think there should always be a "Last" except if there has been no

trading yet that day, on that issue, i.e. if the volume is 0 I think there

will be no Last. TWS itself will display the closing price with a "C"

appended in that circumstance, but the API does not do this.

Of course you can have a real scenario in which there is no Bid because no

one is buying–there is no market, as in the case of options near expiry

that are far out of the money.

-Kurt

[Q] reqMarketData(): sometimes getting back tickPrice events with a price of -1

[A] by Kurt

I have the impression even IB does not know what values will be returned in

the price field. I have variously seen -.99 and -1.0 and 0 and

no-value-transmitted after hours. IB says they pass on data directly from

the exchange (generically speaking) and have no control over it. So you

will get all kinds of nonsense after hours while the exchanges dick around

with their systems. You have to determine yourself whether a price is out

of range. Even during trading hours exchanges may post non-liquid bid/ask

values that look very good but won't execute. For this reason some people

restrict their market data to a single high-volume exchange rather than

using @SMART for equities, even though they still place trades using SMART.

For FX that doesn't apply.

For combos a negative price can (sometimes) be valid, so always good to

examine quotes on all the legs as a reality check. The bid/ask on a native

combo should always be inside what you would compute by adding/subtracting

the quotes you get for the legs, remembering to swap bid/ask when for

negative weights (ratios).

The market data feed you get from IB is kind of a free-for-all. There are

no rules. A price is either real or it isn't. If it's not, it won't

execute. 😉

-Kurt

[A2] by climbon6

IDEALPRO's hours are 17:15 – 17:00 (ET) Sunday-Friday.

The market was closed when you got the Bid price = -1.0

http://interactivebrokers.com/en/trading/exchanges.php?exch=ibfxpro&showcategories=FX&ib_entity=llc

[Q] What is conId?

[A] by Kurt

conId is IB's way of identifying a contract. Think of it as IB's own cusip.

At least AFAIK it has no relation to any one else's system.

There are many contexts in which conId is the only method to identify a

contract. If you specify a contract via the Contract structure, you will

sometimes need to request contract details just to get the conId so you can

use it where it is required. One such place is when building combos. Look

at ComboLeg and you will see there is no other way to specify a leg except

by the conId.

I can't remember whether there are any other contexts that absolutely

*require* conId.

However you might possibly want to use it for persistence of contracts in

use from one invocation of your app to the next, since even if the symbol

for a contract is changed, the conId will not. (This will cease to be the

problem for US options that it used to be, but it can happen with stocks as

well, much more rarely.)

-Kurt

[A2] by Kurt (acutally answer taken from other thread, but this particular one is more about conId):

To the OP's question very likely your point is basically correct. If

setting contract id to 0 worked fine, then there is no (immediate) need to

specify it. (Perhaps now IB has started checking that you don't specify

BOTH the contract id and other Contract fields?)

In any case it would be interesting if the OP would give a full example of a

case that fails, so we can take a look at it.

However I have to disagree re the contract id being only relevant for

combos.

One helpful technique to avoid problems with ambiguous contracts and

Contract specifications that are rejected by requestMarketData or

placeOrder, is to specify the full Contract fields only in

requestContractDetails, obtain the contract id from that, and then use that

contract id (without the other Contract fields) to request market data and

historical data and to place orders.

Various people have been running into problems recently with certain

Contract specifications that used to work and/or that still work with

requestContractDetails but not with requestMarketData or placeOrder. Using

the Contract Id (instead of the other Contract fields) for as many things as

possible avoids this kind of issue, and avoids strange surprises when IB

periodically messes with how placeOrder (or whatever) interprets the other

contract fields.

Using the Contract id also gets you through any symbol changes that might

occur overnight (used to happen regularly with options prior to OSI), which

might find you unable to close the position you opened yesterday using the

same Contract fields.

And of course, as you said, you are forced to use it when specifying combos,

but it is relevant for many other reasons.

-Kurt

[Q] Volume looks incorrect?

[A] by souqmate@yahoo.com

volumes for US stocks have an implicit factor 100 in TWS and in API; true for historical data and for real time data (bidSize or askSize).

[Q] How to get indices historical prices?

[A] by orionn2 <no_reply@yahoogroups.com>

indices don't have bid nor ask prices, hence no (bid ask) midpoint data is available, try using TRADES instead which will give index prices.

[Q] How to get realtime P&L?

[A] by Kurt

> Hi all:

>

> I'm having a very difficult time trying to get realized and unrealized P&L

> from IB. After I establish a connection I call reqAccountUpdates(true, "");

> on the socket, the updateAccountValue() callback method gets called

> frequently, but updatePortfolio() gets called almost never. Is there

> something I'm missing?

I'm not watching that data carefully and since it is the weekend I can't

really go try it. So I'm not sure what your issue is.

My recollection is these things get called periodically, maybe every few

seconds. That's not often enough for my liking. (I think you also get

updates after trades more quickly.)

Personally I like to see P&L in realtime and these updates are not realtime.

For now I use the IB-provided position and cost basis, but I ignore the rest

and compute the P&L myself from the position, cost basis, and tick data.

TWS itself could have done the same, but does not. It seems idiotic to

display delayed P&L values when live data is just sitting there being

ignored. The only reason for TWS to use the delayed P&L data would be as a

fallback if you are not getting IB tick data. But if you are writing an API

app then you don't have that excuse–you can use tick data wherever you get

it from to compute your P&L.

Also if you do it yourself you get to set your own policy on how P&L should

be calculated. I believe IB uses the last price clamped to within the

bid/ask range. That's reasonable perhaps as a general approach but may not

be a safe enough indicator for some people's liking, since the "Last" price

may never happen again. Worse yet if it is delayed a few seconds!

-Kurt

[Q] reqRealTimeBars() – "whatToShow" string?

by someone765@yahoo.com

Recently someone asked if the "whatToShow" string could have multiple entries or would be single valued: TRADES, BID, ASK, MIDPOINT".  Kurt kindly pointed out that the entry had to be single valued.

Do I correctly assume that the realtime bar data will continue with what was last set in this field, until overridden by a new reqRealTimeBars() command? And can a person obtain the trades and bid and ask data successfully during just one 5 second bar, but 3 requests will have to be rapidly made within this interval specifying such to obtain the data successfully?

In other words, can we obtain all the data for the 5 sec bars for a stock and how is this done?

Randall

[A] by Ed news1000@edshome.net

Yes, you request once and then get a reply automatically every 5 seconds.

Each request has its own id so you can have multiple requests concurrently.

You are still bound by the data limits (100 simultaneous market data requests by default) but I'm not sure exactly how this applies to realtime bars. If you are just monitoring a few symbols it will be fine but if you are looking for thousands then it will be a problem.

–Ed

[A2] by Kurt Bigler kkb@breathsense.com

Each request is a subscription, in a sense just like reqMktData.  If you want updates for TRADES, BID, and ASK, you make 3 requests.  People usually make realtime bar requests to get ongoing updates, not for a single bar.  You have no guarantee that your first results for all 3 will start in the same bar period.  But they will continue and you will end up getting ongoing updates from all 3.

-Kurt

[Q] reqRealTimeBars() – only 5sec bars available? Partial bars?

[A] … you can calculate them from 5-second bars. You can calculate other bars from 5sec bars, or you can calculate bars from market data, or you can simply request historical bars periodically, which even lets you get partial bars in progress when you request endDateTime="".

-Kurt

[Q] tickPrice(): canAutoExecute field – wtf?

The documentation on this flag is as follows:

Specifies whether the price tick is available for automatic execution. Possible values are:

• 0 = not eligible for automatic execution

• 1 = eligible for automatic execution

By 'automatic execution' do they mean 'placing an order'? If so, what is the point of sending ticks which are not eligible?

[A] src

My understanding of autoexecution is that it's a hold-over field from the pre-hybrid days…. on the NYSE they used to have two exeuction platforms, one that was manual through the floor specialists and one that was automatic through a computer system… the computer-based orders would only handle a certain volume, so orders that could be autoexecuted and hit on the automated system were tagged for the trader.

However in 2007 they upgraded their automated system and moved everything that way, so I think 99-100% of orders posted to the book should be able to be executed automatically (no human intervention required to hit the bid or take the offer).

So I can ignore it as legacy?

yeah I wouldn't worry about it.

[Q] When requesting historical data I often get “Invalid step”

 What should I do?

[A] "Invalid step" really means invalid combination of step (bar size) and request duration.  Usually you know what bar size you want, without question, and will use whatever is the maximum duration that works, so "Invalid duration for bar size" would be more sensible.  But IB seems to think consistently the other way on this, both in the API errors and in their documentation.

[see table on the next page]


I have always referred to the following table which has been posted to this list at various times, and as a result I have never gotten the Invalid step error.

    http://www.tradingsoftwarelab.com/jtwsdump/table.html

by Kurt Bigler kkb@breathsense.com via yahoogroups.com

mark duration values BLUE which yield max number of bars (multiple marked in one raw – means yield same results aka “same good”)

October 2020: Important update: It seems IB changed (improved) the limits on the historical data requests rate! Not yet documented in IB's documentation.

See Richar's findings in this post:

https://groups.io/g/twsapi/message/45668

16-Feb-2021: Richard shared more details on his finding. Thank you, Richard! I'll quote from "View/Reply Online (#46534)"

Mike

 

For each bar size allowed by the API, there seems to be a maximum 'sweet' duration: if that duration is exceeded, historical data requests take much longer to complete. This must be a policy decision rather than any technical limitation.

 

Below is a table extracted verbatim from my code, which summarises the maximum durations I use for each bar size. These durations were basically determined by trial and error. I don't know whether they're definitive, but they seem to work well for me. Of course there's nothing to prevent IB changing their implementation in any way they see fit at any time, so there's no guarantee that this will continue to be valid forever. Indeed  this table is massively different (and incomparably better) than a similar table I derived when I first started using the historical data api well over a decade ago.

 

Some of the entries in the table may surprise you: for example you can request 50 years (yes that's YEARS) worth of daily data for, say Microsoft, in a single request and the data is returned in less than 5 seconds. Since Microsoft hasn't been around that long, you get all the data IB has for them, which goes back to 1986: here's the very first bar, for 13 March 1986:

 

Bar date=19860313;Open=28.00;High=29.25;Low=25.50;Close=28.00;Volume=35826;WAP=28.002;Tick volume=1;

 

I used to swear at IB's historical data servers, but I'm very impressed with the performance now (though I think the API itself is a monstrosity).

 

So here's the table:

 

'   Bar Size            Max Duration

'   ——–            ————

'

'   1 secs              2000 S

'   5 secs              10000 S

'   10 secs             20000 S

'   15 secs             30000 S

'   30 secs             86400 S

'   1 min               86400 S

'                       6 D

'                       1 W

'   2 mins              86400 S

'                       10 D

'                       2 W

'   3 mins              86400 S

'                       10 D

'                       2 W

'   5 mins              86400 S

'                       20 D

'                       3 W

'   10 mins             86400 S

'                       50 D

'                       8 W

'   15 mins             86400 S

'                       50 D

'                       10 W

'   20 mins             86400 S

'                       50 D

'                       10 W

'   30 mins             86400 S

'                       50 D

'                       10 W

'                       3 M

'   1 hour              86400 S

'                       50 D

'                       10 W

'                       3 M

'   2 hours             86400 S

'                       50 D

'                       10 W

'                       3 M

'   3 hours             86400 S

'                       50 D

'                       10 W

'                       3 M

'   4 hours             86400 S

'                       50 D

'                       10 W

'                       3 M

'   8 hours             86400 S

'                       50 D

'                       10 W

'                       3 M

'   1 day               86400 S

'                       365 D

'                       12 M

'                       52 W

'                       50 Y

'   1 W                 86400 S

'                       365 D

'                       12 M

'                       52 W

'                       50 Y

'   1 M                 86400 S

'                       365 D

'                       12 M

'                       52 W

'                       50 Y

 

Richard

\ duration

   \

     \

        \

bar size\

Y

M

W

D

S

1 year

86400

3 month

86400

1 month

5

86400

1 week

5

12

52

60

86400

1 day

5

12

52

60

86400

1 hour

1

4

34

86400

30 min

1

4

34

86400

15 min

2

20

86400

5 min

10

86400

3 min

10

86400

2 min

10

86400

1 min

10

86400

30 sec

1

86400

15 sec

30000

5 sec

10000

1 sec

2000

Return to the jTWSdump homepage.

Example: I want to pull bar size = 30 min, and I’m going to specify duration in “D”ays, which value for duration should I use (i.e. what is tha max allowed one?). The table yelds “34”, so the answer is “34 D” for your duration parameter for reqHistoryData().

Note, 34 trading days is > than 34 calendar days.

Note: 34 trading days is > than specifying “1 W” or “86400 S”.

(todo: run all those queries and add into the table actual number of bars returned for cases where bar size != duration units, mark duration values BLUE which yield max number of bars)

also: Riggster: I didn't realize that when you specify a barsize of 1 – 15 seconds, you have to specify the durationStr as seconds, not days, so you could use '23400 S', but not '1 D', even though there are 23400 seconds in 1 day.

[A2] by orionn2 (18-Dec-2015):

I have updated the table in the jTWSdump website: http://www.tradingsoftwarelab.com/jtwsdump/table.html

showing increased limits of 5 Y (5 years) for 1M (1 month), 1W (1 week) and 1 day bars and also 10 D (10 days) for 5 mins, 3 mins, 2 mins and 1 min bars. These limits are for a single data request – much more data can be downloaded by setting the End Date/Time in another API request to a date before 5 years from now. In jTWSdump this can be easily managed by using the "Number of Queries" field.

Thanks to Richard L King for the heads-up.

also see next Q/A about bar size settings:

[Q] Valid Duration and Bar Size Settings for Historical Data Requests

[A] quote from this page:

The following table lists valid duration and bar size settings for API historical data requests. Please note that these are only guidelines.

Duration

Bar Size

DURATION

1 Y

BAR SIZE

1 day

6 M

1 day

3 M

1 day

1 M

1 day, 1 hour

1 W

1 day, 1 hour, 30 mins, 15 mins

2 D

1 hour, 30 mins, 15 mins, 3 mins, 2 mins, 1 min

1 D

1 hour, 30 mins, 15 mins, 5 mins, 3 mins, 2 mins, 1 min, 30 secs

14400 S (4 hrs)

1 hour, 30 mins, 15 mins, 5 mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs

7200 S (2 hrs)

1 hour, 30 mins, 15 mins, 5mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs

3600 S (1 hr)

15 mins, 5 mins, 3 mins,  2 mins, 1 min, 30 secs, 15 secs, 5 secs,

1800 S (30 mins)

15 mins, 5 mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs

960 S (15 mins.)

5 mins, 3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs

300 S (5 mins)

3 mins, 2 mins, 1 min, 30 secs, 15 secs, 5 secs, 1 secs

60 S ( 1 min)

30 secs, 15 secs, 5 secs, 1 secs

[Q] Is it possible to develop TWS like depth book using reqMktDepth() call?

by hyeungsf@yahoo.com

[A] by barnstws <no_reply@yahoogroups.com>

Try specifying the primary exchange field. reqContractDetails will help you get the exact parameters. Forex is free and easier to see if you are doing it right.

(update 04-Feb-2021) But note: with standard 100-lines subscription you can run only 3 reqMarketDepth subscriptions at a time (only 3 symbols, not 100 of them).

[Q] implementation notes: reqMarketDepth

[A] by Dmitry

Added on 04-Feb-2021

Looking at reqMarketDepth documentation

was not clear what the "isSmartDepth" is there for:

Another issue in the documentation is: "This request must be direct-routed to an exchange and not smart-routed."  <– this can be a problem since we use SMART routing and do not want/need to know exact exchange (often times it can be many, right?)    (also SMART routed orders are cheaper than direct routed, so we should stick to SMART).

googling for: tws api isSmartDepth

revealed this nice discussion (on my favorite forum)

which leads to this notes on the arguments (from the release notes for version 974):

isSmart Depth: The API now provides aggregated depth of market (DOM) quotes from the level 1 and level 2 feeds to which a user has subscribed, instead of requiring the API client to make reqMktDepth requests to each exchange individually. This is requested by setting the new parameter isSmartDepth in reqMktDepth to True.

Also forum  have another nice note:

mktDepthOptions: is reserved for internal use and should just be left null

Great! So we keep "mkdDepthOptions = null" all the time and "isSmartDepth = true", this way we shouldn't care about direct routing! Huray!

Now looking at ApiController.java file in the speculant platform (this file implemented by IB) we see:

1) we only need to send reqDeepMktData() (check this out – we can even make links to certain line numbers in the code!!  line 648)

2) responses from both updateMktDepth() and updateMktDepthL2() will go to our the only handler, which we need to implement (function updateMktDepth() the only member of interface IDeepMktDataHandler)

and that is it!

Now all you need is to store incoming results callbacks from IB API.

I've decided to store them in 2 HashMaps "tables": bid_table and ask_table (so we are ok with any order of updates and even with gaps if they say "DELETE" record from the middle of the table, which is completely possible.. SimpleArrayList would not allow to do so. Now position is the key of the hashmap and the value is a class (read "structure") holding all incoming arguments together (and I added timestamp into it's constructor, to know exactly when this row was received (with "ms" resolution)).

[Q] What happens when I do reqMktData() afterhours?

[A] by Dmitry

In case of successful subscription you’ll get one tickPrice() event with tickType=CLOSE (basically latest price (no volume))

OR

You’ll get either an error() event (if your contract description is somewhat broken or you set same tickerId as for your previous subscription or other error)

Examples:

[successfull afterhours subscription]

05:18:20 PosixTestClient.cpp:739 reqMktData():

05:18:20 PosixTestClient.cpp:746 requesting market data for contract:

             conId: 0

            symbol: FB

           secType: STK

            expiry:

            strike: 0

             right:

        multiplier:

          exchange: SMART

   primaryExchange:

          currency: USD

       localSymbol:

      tradingClass:

    includeExpired: no

         secIdType:

             secId:

  comboLegsDescrip:

         comboLegs: no legs found

05:18:20 PosixTestClient.cpp:247 tickPrice():

         tickerId: 0

         TickType: CLOSE (9)

         price: 55.44

         canAutoExecute: 0

[bad afterhours subscription attempt]

05:23:15 PosixTestClient.cpp:739 reqMktData():

05:23:15 PosixTestClient.cpp:746 requesting market data for contract:

             conId: 0

            symbol: FB

           secType: STK

            expiry:

            strike: 0

             right:

        multiplier:

          exchange: SMART

   primaryExchange:

          currency: CAD <–– this makes it bad

       localSymbol:

      tradingClass:

    includeExpired: no

         secIdType:

             secId:

  comboLegsDescrip:

         comboLegs: no legs found

05:23:15 PosixTestClient.cpp:232 error():

         id: 0

         errorCode: 200

         errorString: No security definition has been found for the request

or some other errors, like:

         errorCode: 322

         errorString: Error processing request:-'vc' : cause – Duplicate ticker id

[Q] IB's own historical data corrections (history of the history:)

Added on 27-Apr-2015

[A] by Kurt (found here)

Although I'd heard it mentioned (even by IB I believe) I had never looked at

what changes over time actually occur in IB's historical data, i.e. the

history of the history.

Today I observed the following in 1-min USD.CHF BID_ASK bars. The times

shown are US Pacific Time.

Some time on 5/30 I downloaded this bar from the day before:

20110529 22:52:00 O 0.85215 H 0.85225 L 0.8521 C 0.85225

but today (6/5) when I download the same bar it had changed to:

20110529 22:52:00 O 0.85215 H 0.85215 L 0.8521 C 0.85215

^ ^

I have not gotten around to comparing this to 1-second data to see how it

relates. I never acquired the data I would have needed to do that in this

case.

(ASIDE: These are BID_ASK bars, so open and close don't have the usual

meaning. However they are supposed to be time-weighted averages, and as

such they are suspect being so exact. I believe it may have come up that IB

rounds their averages to the nearest tick, which of course makes the average

less accurate. I think I had observed this but also I had some other

problems that may have occluded accurate observation, since I was

unknowingly rounding my own archived data (to what the C format %g

produces). Since fixing that I have observed that OPTION_IMPLIED_VOLATILITY

historical data does NOT have the time-weighted averages rounded. But of

course there is no minTick value that applies to volatility.)

I also have seen evidence of some BID and ASK bars disappearing from the

record. These may have been after-hours bars. I don't have the info handy

on this right now.

The main point is that I have now seen an actual change in the history of

historical data, whereas previously this was just conjecture for me. I

don't know exactly when the change noted above was made because it was a

week before I checked it again.

I'll eventually look at this in more detail, but I'm wondering what others

do with respect to this issue.

One possible strategy is to wait a certain period of time (several days?)

for the "majority" of history changes to settle out, before downloading.

I'm curious how many people intentionally do NOT download up to the current

moment for this reason. If so, how long do you wait for data to stabilize?

If you answer, please indicate what security or type of security you are

working with.

Thanks for any thoughts.

-Kurt

[Q] can reqTickByTickData get delayed ticks?

[A] by josh View/Reply Online (#44949) 

Added on 28-Jul-2020

The different data types including delayed data are only available with the reqMktData function.

reqTickByTickData can only be used with live data which requires market data subscriptions for most instruments.


TWS Scanners

[Q] Where do I begin with scanners?

[A] Look at the Market Scanner item on the Market Data tab of the ApiDemo program. This will give you a good start.

[Q] Biggest gainers since open between $5 and $10?

Trying to return the biggest gainers since open between $5 and $10.

 I have successfully returned this in TWS, but I cannot get the exact same result in the API. Below is my code, then the TWS scanner file that I am trying to copy.  What am I missing here?  Thanks!

Creating scanner:

ScannerSubscription scan = new ScannerSubscription();

            scan.scanCode("TOP_OPEN_PERC_GAIN");

            scan.instrument("STK");

            scan.locationCode("STK.US.MAJOR");

            scan.belowPrice(10);

            scan.numberOfRows(10);

            

            m_client.reqScannerSubscription(0, scan);

Output method:

public void scannerData(int reqId, int rank, ContractDetails contractDetails, String distance, String benchmark, String projection, String legsStr) {

            System.out.println(contractDetails.m_marketName + " " + reqId + " " + rank + " " + distance + " " + benchmark + " " + projection + " " + legsStr);

    }

The TWS file:

<?xml version="1.0" encoding="UTF-8"?>

<ScannerTemplate autoRefresh="true">

        <ScannerContent varName="content" version="0" watermarkWhenEmpty="false" pageId="1385043649" transferableParameter="true" locationText="STK.US.MAJOR" statsExchText="ALL" maxItems="-1" snapshot="-1" paused="false" scannerSent="true" chartDividerLocation="0">

                <ExtendedConfigurationMap varName="extendedConfiguration"/>

                <scannerName></scannerName>

                <Instrument varName="instrument" name="US Stocks" m_type="STK" filters="PRICE,PRICE_USD,VOLUME,VOLUME_USD,AVGVOLUME,AVGVOLUME_USD,HALTED,UNSHORTABLE,SHORTSALERESTRICTED,NUMSHARESSHORT,SHORTOFFLOATPERC,DAYSTOCOVER,NUMSHARESINSIDER,INSIDEROFFLOATPERC,NUMSHARESINSTITUTIONAL,INSTITUTIONALOFFLOATPERC,OPTVOLUME,AVGOPTVOLUME,OPTVOLUMEPCRATIO,IMPVOLAT,IMPVOLATOVERHIST,MKTCAP,MKTCAP_USD,STKTYPE,PILOT,DIVIB,DIVYIELDIB,NEXTDIVDATE,NEXTDIVAMOUNT,HISTDIVIB,HISTDIVYIELDIB,GROWTHRATE,PERATIO,QUICKRATIO,DIVYIELD,RETEQUITY,PRICE2BK,PRICE2TANBK,CHANGEPERC,AFTERHRSCHANGEPERC,CHANGEOPENPERC,OPENGAPPERC,PRICERANGE,IMBALANCE,IMBALANCEADVRATIOPERC,REGIMBALANCE,REGIMBALANCEADVRATIOPERC,AVGRATING,NUMRATINGS,AVGPRICETARGET,NUMPRICETARGETS,AVGTARGET2PRICERATIO,TRADECOUNT,TRADERATE,VOLUMERATE,SIC,INDEX_COMPARISON,FIRSTTRADEDATE,HASOPTIONS" group="STK.GLOBAL" shortName="US"/>

                <ScanType varName="parameter" displayName="Top % Gainers Since Open" scanCode="TOP_OPEN_PERC_GAIN" instruments="STK,STOCK.NA,STOCK.EU,STOCK.HK,FUT.US,FUT.HK,FUT.EU,FUT.NA" absoluteColumns="false" supportsSorting="true" respSizeLimit="#" snapshotSizeLimit="250" feature="PCT" searchDefault="false" access="allowed">

                        <Columns varName="columns">

                                <Column colId="55" name="Change Since Open" display="true" section="m">

                                        <displayType>DATA</displayType>

                                </Column>

                                <ColumnSetRef colId="0" name="BA" display="false">

                                        <displayType>DATA</displayType>

                                </ColumnSetRef>

                                <Column colId="48" name="Time in Scan" display="true">

                                        <displayType>DATA</displayType>

                                </Column>

                        </Columns>

                </ScanType>

                <SettingValueList varName="settingList">

                        <SettingValue settingCode="annualVolatility" value="true"/>

                </SettingValueList>

                <AdvancedFilter varName="filter">

                        <priceAbove>5</priceAbove>

                        <priceBelow>10</priceBelow>

                </AdvancedFilter>

        </ScannerContent>

        <ArString varName="fullySelectedNodesList">

                <String>STK.US.MAJOR</String>

        </ArString>

        <Columns varName="tickerColumns" type="jcolumn.TickerColumns">

                <ContractSingleDescColumn colId="99" width="176" alignment="2" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false"/>

                <ChangeSinceOpenColumn colId="55" width="172" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" showPercent="true"/>

                <LastPriceColumn colId="7" width="188" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" prohibitDynamicColors="false"/>

                <BidPriceColumn colId="4" width="188" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" prohibitDynamicColors="false"/>

                <AskPriceColumn colId="5" width="256" alignment="4" fg="-1" bg="-16777216" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false" prohibitDynamicColors="false"/>

                <InScanTimeColumn colId="48" width="256" alignment="4" uniqueId="0" maxSymbolWidth="0" overrideBg="false" overrideFg="false"/>

        </Columns>

</ScannerTemplate>

Oh, and when I create the scanner, there should be a line that says:

scan.abovePrice(5);

Idk why that didn't copy over.

[Q] Market Scanner Pre/Post Market Hours

[A] by Josh

Added on 18-Dec-2021

From: https://groups.io/g/twsapi/topic/29078601#41570

Hi Nick,

 

If you receive scan results in the TWS Advanced Market Scanner which are not available to the API, this can be because the results contain contracts which are not tradable through TWS. The TWS Advanced Market Scanner has a feature to automatically skip these results, whereas the API scan does not. I would suggest first double-checking what the TWS Advanced Scanner displays for the same scan if the scan code and filters are chosen to match exactly.

 

Josh (IBKR)

Thank you Josh, I found out that if I place a condition for minimum volume, the scanner will work as expected. subscription.MarketCapAbove = 1e6;

NickF

Also: by David Armour View/Reply Online (#41903)

OK – I figured out a solution with the scanner to get what I need for stocks that are gapping in the pre-market..

I use Top % Gainers/Losers (and not Close to Open)

I use Change % (and not Gap %) ensuring to set LESS THAN -X% when looking for losers and GREATER THAN X% when looking for gainers.

If I set the filter to the Change % I get the Gap % since the close of the previous day. This is working in Pre-Market which is what I needed.

I hope this helps anyone else struggling with pre-market scans.

[Q] Trouble getting API scanners to output the same date as the one in TWS!

From: bgould96@aol.com

Sent: Sunday, November 24, 2013 11:19 AM

To: TWSAPI@yahoogroups.com

Subject: [TWS API] RE: Trouble getting API (Java) scanner to output the same data as the one in TWS

[A] by WeifeiL@gmail.com weifeil@gmail.com via yahoogroups.com

Two things:

1. there is a bug for API:

 

you need to add this one to the scan obj:

 

scan.averageOptionVolumeAbove( 0 );

 

 

2. For each scan, you need to specify a unique id. And this id will be associated to the scanner and not changeable before you restart your IB TWS/Gateway.

 

Good luck with the scanner functions.

 

Weifei

[A2] by supermanz@gmail.com via yahoogroups.com

I think you're looking at the wrong field in the result.  Try contractDetails.m_summary.m_symbol.

bgould96@aol.com via yahoogroups.com:

Ahhhh, that works!  Thank you so much!  Just purely for the sake of knowledge, what exactly is returned by contractDetails.m_marketName?

[Q] Give me simple example of working scanner request.

[A] you can get it from following communication (taken from here):

I have the code for connecting to TWS … it's been working for several years. Today, I tried to access the scanner … like this:

… connect to TWS

ScannerSubscription ss = new ScannerSubscription();

ss.numberOfRows(10);

ss.instrument("STK");

ss.locationCode("STK.NYSE");

ss.abovePrice(1.0);

ss.belowPrice(10.0);

ss.scanCode("TOP_PERC_LOSE");

ss.stockTypeFilter("STOCK");

toTws.reqScannerSubscription(id, ss);

… after scannerDataEnd is called, I can successfully cancel the scanner subscription

When I run the above I get:

error: id: 936, errorCode: 165, msg: Historical Market Data Service query message:no items retrieved

If I interchange 1.0 and 10.0 above, I get:

error: id: 936, errorCode: 162, msg: Historical Market Data Service error message:Invalid Price Filter

Clearly I'm missing something or I don't understand something. Can anyone provide any insights???

———————————— answer ———————

Hi Lewis,

From ScannerSubscription.java:

private double m_abovePrice = Double.MAX_VALUE;

private double m_belowPrice = Double.MAX_VALUE;

private int m_aboveVolume = Integer.MAX_VALUE;

private int m_averageOptionVolumeAbove = Integer.MAX_VALUE;

private double m_marketCapAbove = Double.MAX_VALUE;

private double m_marketCapBelow = Double.MAX_VALUE;

Set abovePrice, aboveVolume, and marketCapAbove (to Double/Integer.MIN_VALUE, for example), and I think your

problem will be fixed.

David

———————————— and final working scanner is:  ———————

Thanks for the help. It works now. Here's the code:

ScannerSubscription ss = new ScannerSubscription();

ss.numberOfRows(10);

ss.instrument("STK");

ss.locationCode("STK.NYSE");

ss.abovePrice(3.0);

ss.aboveVolume(0);

ss.marketCapAbove(100000000);

ss.averageOptionVolumeAbove(0); // had to have this!!

ss.scanCode("MOST_ACTIVE");

toTws.reqScannerSubscription(936, ss);

[Q] Can't get scannerSubscriptionOptions to work

Is anyone else using scanners?  They are so under documented!  I am doing bond scans which work fine in part.  For example (in C# calling J#):

                int nTickerID = 301;

                ScannerSubscription ss = new ScannerSubscription();

                ss.numberOfRows(10);

                ss.instrument("BOND");

                ss.locationCode("BOND.US");

                ss.scanCode("HIGH_BOND_NET_ASK_YIELD_ALL");

                ss.maturityDateAbove("20171001");

                ss.belowPrice(100.1);

                ss.moodyRatingAbove("BA3");

                ss.spRatingAbove("BB-");

             

                java.util.Vector opts = new java.util.Vector();

                opts.add(new TagValue("VARCPNRATE", "false"));

                engine.IBHandler.ClientSocket.reqScannerSubscription(nTickerID++, ss, opts);

I'm returned the 10 bonds I requested, except I can't get scannerSubscriptionOptions to work for me i.e. they are a mix for fixed and variable rate bonds.  I have studied the reqScannerParameters() XML to try to filter down more.  VARCPNRATE is supposed to work for BOND, but how?

Has anyone gotten scannerSubscriptionOptions  to work?  Is there a guide anywhere for these TagValue options?  Any hints are appreciated!

[A] by Josh (found here)

Added on 10-May-2017

Hi Harlan, unfortunately the API scanner is quite limited in available filters and scans as compared to the TWS Advanced Market Scanner. The reqScannerParameters has parameters which are available in the TWS scanner but not all are available from the API. The third parameter in reqScannerSubscription is used only by some third party applications to specify the origination of the request, otherwise its null. The available filters are simply the members of the ScannerSubscription object.

Josh

[Q] Note on contracts which may not be accessible through TWS when running scanners:

[A] by Josh View/Reply Online (#41572)

If you receive scan results in the TWS Advanced Market Scanner which are not available to the API, this can be because the results contain contracts which are not tradable through TWS. The TWS Advanced Market Scanner has a feature to automatically skip these results, whereas the API scan does not. I would suggest first double-checking what the TWS Advanced Scanner displays for the same scan if the scan code and filters are chosen to match exactly.

[Q] Market Scanner Pre/Post Market Hours:

[A] by David Armour dave.armour    at  gmail.com View/Reply Online (#42026)

OK – I figured out a solution with the scanner to get what I need for stocks that are gapping in the pre-market..

I use Top % Gainers/Losers (and not Close to Open)

I use Change % (and not Gap %) ensuring to set LESS THAN -X% when looking for losers and GREATER THAN X% when looking for gainers.

If I set the filter to the Change % I get the Gap % since the close of the previous day. This is working in Pre-Market which is what I needed.

I hope this helps anyone else struggling with pre-market scans.

  • To clarify, will the Top % Gainers/Losers scanner update pre/post hours, without any filters ?
  • Yes. Just don't use the After-Hours or Pre-Market Hours option which seems only to work when the market re-opens.

[A2] by Josh View/Reply Online (#41888):

On Sat, Mar 2, 2019 at 05:26 PM, Lightbrook Brokers wrote:

Will the callbacks for reqMktData (of tick type 4, last price) continue updating for pre/post hours?

Yes the results including last traded price will continue to update in pre/post regular trading hours.

I also tried Top % After Hours Gainers/Losers and that also returns nothing.

You might want to take a look at this post https://groups.io/g/twsapi/message/41570 and add a filter in the same way to limit the scan results further. If the scan results are instruments which can't be traded at IB they won't be returned to the API.

[Q] How to find the location code for Australian when creating a scanner subscription object?

[A] View/Reply Online (#45084)

Added on 08-Feb-2021

This is the method which I use to find the "Location Code".

1) Make a call to reqScannerParameters

2) In the call back scannerParameters. you will receive  a string containing all available XML-formatted parameters.

3) Open the XML file and make a search for "Australia". See the attached figure for details.

4) Punch in the Location Code from this XML into your python program for ScannerSubscription object.

scanSub = ScannerSubscription() # Defines a market scanner request

scanSub.instrument = 'STOCK.HK'

scanSub.locationCode = 'STK.HK.ASX'

Rgds.,

Pavan

IB News subscription

[Q] When subscribing to a news feed via API, is it possible to receive the rank?  If yes, how?

[A] View/Reply Online (#45989)

Added on 13-May-2021

– I'm just guessing but it looks like it's encoded in the extraData argument passed to tickNews().

– I confirm, but be careful that is a string and not documented.

– Thanks, guys!  I see it in the extraData – something like – "L:unknown:K:-1.00"   Where "K" must stand for rank, but some messages don't have it at all, so being careful is good advice.

Basics for beginners

You should always try the corresponding functionality in TWS. If it

doesn't' work in TWS either, then it is not an API problem. (Kurt)

[Q] Vocabulary

[A]

ATS – automated trading system

BBO – Best bid offer (NBBO = National BBO) (RBBO = Regional BBO)

DOM – Depth of market

CFD – Contract for Difference

GAT – good-after-time (GAT) orders

MOC – market on close order (example: MOC order in an oca group)

              MOO and MOC work for U.S. stocks. Check the IB website for other markets.

              One catch for MOO orders is that you have to specify "OPG" for the time-in-force.

Also, these order types can not be canceled after a specific time. I

haven't done this is in a while but it used to be for stocks you could

not cancel a MOC after 3:50 pm NY time. Be sure to research all the

details before you place these order types.

MOO – market on open order

OCA – once-cancels-all

OCO – one-cancels-other (when one order executes it will automagically cancel another order:)

Note about OCA/OCO: by Richard L King from this thread

There is no guarantee that this will avoid the problem. Where two orders in an OCA group have fairly close prices, it is possible for both to be filled in a fast market. This is because when the first order is filled, IB cannot know to cancel the second order until it has received the fill notification for the first from the exchange, and it then has to transmit the cancellation to the exchange: during this time the second order could be filled.

This did actually happen to someone a few years ago, and they complained to IB, but IB (quite rightly) pointed out that they cannot make any guarantee that only one order in an OCA group will be filled.

A much safer approach is to simply use a single stop loss order, and adjust its price as required: it can be included as a child order when the entry order is submitted, and with a wide stop for safety. Once the entry order has filled, the trigger price for the stop loss order can be tightened as required, and subsequently adjusted as profit increases.

RTH – regular trading hours (like 9:30 to 16:00)

NBBO – Relative (a.k.a. Pegged-to-Primary) orders provide a means for traders to seek a more aggressive

price than the National Best Bid and Offer (NBBO)

VWAP – volume weighted average price

[Q] Limitations on quantity of symbols?

[A] from IB website

All customers initially receive 100 concurrent lines of real-time market data (which can be displayed in TWS or via the API) and always have a minimum of 100 lines of data. Customers can increase the allowance of simultaneous quotes windows by purchasing monthly Quote Booster packs.

[Q] Where can I find the API version info?

[A]

C:\Jts\API_VersionNum.txt

[Q] I’m about to post a question to IB API list. What should I do?

[A]

1. do basic google search for your question before asking your [Q]

2. search TWS API archives for some of keywords related to your question

3. sure, nothing found.. you’re writing your post to The Group

    – include the API type that you trying to use

    – try to include what exactly you’re sending to API

    – what replies (in form of callbacks) you receive?

        – are you logging all the possible API callbacks or may be you just missing one

   thus thinking “I did ABC, but nothing happened…”?

    – generally try to be precise (not vague)

    – try not to send more than 10 posts/day 😉

     (Few thousands subscribers will have to read it. Have some respect:)

[Q] Number of hops to IB?

[A]

Has anyone else experienced HDMS market data connection issues since upgrading

to 906?

I couldn't get a stable connection so I downgraded back to 900.

Support is telling me I have too many hops in my tracert! Apparently the support

engineer thinks British Telecom will reduce them for me down to 15 (from 26) if

I complain, or else change ISP!

— Adam Hardy

I always used to ping gw1.ibllc.com

Right now I get about a 50 msec avg round trip from North Carolina, USA on

Road Runner Turbo over 14 hops.

— FutureScalper

others:

14 hops, 55ms here (16Mb comcast business class)

9 hops, 37ms (Small Co-Location facility in Maryland)

In relation to connection speed, just reading the other current thread, tracert

shows I am 25 hops from the server at 200ms – I'm in Europe – and the IB support

staff say this is not a guaranteed good connection 🙁

Ping @ 95ms each.

[Q] Is IB real time data actually “real time data”?

[A] (investigate more)

IB's data is not "live" to begin with. it's series of snapshots with 5-30 ms intervals(don't remember exactly, you can check TWSAPIgroup on yahoo-there is very detailed explanation somewhere about how IB deliver the data). bottom line-it's not live and this is why IB's data usually doesn't lag at the open or during Fed announcements. so-there is no such thing as "fast market" for IB's data

— also —

You would not expect IB's ticks to match anyone else's. You are aware that

IB's ticks are sampled?

-Kurt

I don't keep the exact details of it in my head, but it is something like a

limit of 10 ticks per second that is imposed, with a higher rate I think for

certain types of instruments. (Someone posted definitive information on

this a few months ago but I can't find the message.)

-Kurt

— also —

IB feed have a high enough resolution to do some meaningful work..

— also —

But if you need unsampled tick data you need to go elsewhere to get that

data.

-Kurt

— also —

this is an excerpt from emini-watch.com :

"The IB data feed available via their Trader Workstation Software (TWS) is not a true tick-by-tick data feed. IB provides snap shots of the trade data several times a second with an aggregate of the trades that took place during that interval. As a result time-based charts (e.g. 5 minute charts) will be correct, however, tick charts will not."

— also — by rwk0434 at gmail.com (src):

There are actually three ways to get data that may be creating some confusion: 1) reqMktData(), 2) reqHistoricalData(), and 3) reqRealTimeBars().  Historical data comes from a different server farm, posing an obvious source of discrepancy.  I am not sure whether realtime bars are built on the primary data server or the historical server, but they appear to have fewer data discrepancies for some applications.

It is well-know that IB data is sampled. (Also note: Stock data is sampled all 250ms and options data all 100ms.) The sampling rate is about 3/second, except for some GLOBEX markets where it is 10/second.  Less well-known (to me anyway) is that some venues have started distributing conflated data.  If you trade stocks and have been following the HFT saga, you probably know that there are data feeds that run ahead of the official SIP feed.

This brings up the question of how usable the IB data is for active trading.  It's my belief that we can have completeness or timeliness, but not both.  IB has chosen timeliness, and that is my preference as well.  If you need completeness, you should consider other data feeds.  If you want both, you will need expensive infrastructure such as co-location.

For traders organizing their data into bars, the effect of sampled data is that occasionally a bar will be missing a high or low, usually by just one tick, though it could be more.  Many traders find that unacceptable.  I am skeptical that the data discrepancy will affect long-term profitability very much.  It's hard to imagine how avoiding these occasional data discrepancies would justify the added cost of more extensive infrastructure.

[Q] When I pass a BUY LIMIT order on the ES, let's say at 1300.25, I always have to wait until the price goes to 1300 to get a fill…is that normal?

[A] by rwk 2095 from this thread

I'm not an expert on the ES, but I will try a comment anyway. When you are using an order type that is native to the exchange ('limit' is native to Globex), there is almost no difference from one broker to another unless you are doing high-frequency trading (HFT). Your order goes to the back of the queue. For a very busy market such as ES, you won't get a fill very often without the inside price moving past your order.

If you're using a non-native order such as 'stop', your order is simulated on the broker's server, and delays can be quite substantial. It's best to avoid non-native orders for that reason.

If you're trading a market that doesn't have a central limit order book, such as the U.S. stock market, there is a lot of difference in quality of fill among brokers. IB has a generally good reputation.

[rwk]

[A2] by poch32

Keep in mind that each DOM level has a FIFO queue. If there are, let's say, 50 contracts for sale @ DAX 7002.5, and yours happens have arrived in place #48, you can easily see that – in practice – you will need the price to completely fill the whole 7002.5 level and go down to 7002.0 to garantize yourself a fill.

[Q] Will I have this problem if I try my algorithm with a stock like IBM on SMART?

[A] by Eric J. Holtman from this thread

It will be infinitely worse with stocks.

At least with the Emini, you've got one central computer

doing the order matching, and assuming everything is

fair, you just wait your turn and get a fill.

Here's what will happen with, say, IBM@SMART.

NBBO is 104.14 x 104.16. Let's say you put in an

order to buy @ 104.14. IB needs to pick one of the

many exchanges to send your order to.

Suppose it picks wrong. It's possible that your order

won't get filled, even if the stock trades all day

at 104.14.

It's also possible for you to not get a fill if the

stock trades at 104.13. Now, in that case, you can

try to file a report with IB. They'll go look at

tapes (if you're lucky), and you *might* see a price

improvement or a fill.

You might not, too. You'll be told any number of

excuses, but bottom line is, sometimes, you just

won't get a fill you think you "deserve".

Now, this isn't just small retail guys getting clipped

for a penny on 100 lots. I used to work for a fund

where we were throwing around tens of thousands of

shares, we had prime broker relationships with big

Wall Street banks, we had >$100mm AUM, and we still

got hosed by "trade throughs" all the time.

It's a price of doing business.

And all the above is just related to actual "trade

throughs", which should be illegal (or quickly fixable)

but in practice, you just have to eat them. On top

of that is just everyday, garden variety slippage:

You see IBM at that same 104.14×104.16. You submit

a market order, expecting 104.16. You get 104.18.

Welcome to trading!

[Q] I am behind a firewall I like to know the TWS out bound ports?

[A] by Kurt

Googling "tws ports" takes me to this IB page:

http://www.interactivebrokers.com/en/general/education/faqs/technical.php

They mention 4000 but I seem to recall I had to open 4000 and 4001 to get it

working. If you get it wrong you will find out quickly.

However there are other ports for things like news feeds. If you run

netstat with the right args (netstat -a I think) you will see all these

things and as I recall they were all pretty obvious from the url that

appears in the listing.

-Kurt

[Q] Other TWS API FAQs?

[A]

https://www.interactivebrokers.com/en/general/education/faqs/technical.php

[Q] Any general recommendations?

[A] by Steve Novotny

I can save you a lot of time: people who want to constantly babble online

about their systems don't have one (which profits, anyhow). People who want

to talk constantly about this and that indicator, and pivots, and technical

analysis, and blah blah blah have never made a nickel trading.

..and people who HAVE come up with a way to profit while sleeping aren't

telling anybody ANYTHING. (the exception to the last two paragraphs may be

those who have found something that, by coincidence, works for AWHILE-long

enough for them to get excited J )

You will save yourself a TREMENDOUS amount of time if you read NO BOOKS

about trading, join NO GROUPS discussing trading, and buy NO MATERIALS

related to trading. Instead, look at historical prices a LOT, brainstorm

about how one might win (which, I can assure you, will be NOT OBVIOUS OR

SIMPLE), then TEST your ideas in a statistically responsible manner.

— also —

Hi Regis,

You might want to join Elite Trader forums. They talk about such things all the time.

Also, if you want a more mathematical insight into things, look at QUANT Finance forums such as Wilmott, or http://www.quantitativefinanceforum.com/

For books, start with Irene Aldridge's book on "High Frequency Trading" and Ernie P. Chan's "Quantitative Trading". Both are quite Easy reads.

You may also want to look into "Statistical Arbitrage" by Andrew Mole.

These should be a good starting point for you.

Steve.

[Q] IB commission rate comparison

[A] https://www.interactivebrokers.com/en/?f=commission#options-clear

[Q] How to sync market data?

by israelieyal: hi all,

i'm using IB to capture real time market data.

right now, i keep it simple and only use reqMktDataEx and listen to the results on the tickPrice and tickSize event Handlers.

this work fine.

my question is how do i know all data (bid, ask, last etc.) is synchronized? for a given time, how do i know all these components (bid, ask, last – sizes and prices) are the actual values for that time?

i'm sure that's a question that a lot of programmers were faced into. i'll be glad to hear your ideas/opinions.

thanks

[A] by Eric J. Holtman

First, the IB data is sampled, so it's *never* actually correct.

Second, there are no timestamps delivered with the data, so there

is no way to tell if it's a late report from an exchange.

Third, quotes can be quite old…. if a stock is thinly traded,

you may receive any updates for seconds, or even minutes.

— Eric

[A2] by skunktrader2001

They are not synchronized. The feed is designed to update a grid.

[Q] Notice of High Order Ratio

[A] by Orionn

just got this from IB:

"Dear Trader,

We have noticed a very high ratio of orders and order modifications

relative to the number of executions in your account ******. Every order

instruction submitted by you (includes new order submission, order

modification, cancellation) utilizes computing power. Excessive order

activity can slow down IB systems and negatively impact other clients.

IB monitors order/trade ratios to prevent excessive and unnecessary

system resource utilization. As a general rule, a ratio value less than

10 order actions per 1 execution will generally be acceptable, although

fees may apply for certain markets where the ratios exceed 5:1. Above

20:1, IB will request clients using automated order management tools to

optimize their order management behavior. Above 100 orders per

execution, or if our analysis indicates a systematic misuse of IB's

automation services, IB may take steps to reduce the system utilization

including: charging a fee for order modifications (typically 20% of the

commission for an execution), or limiting access to API services.

Most of the time, excessive order/trade ratios are caused by poorly

optimized APIs or other order creation programs. These applications

generate price changes that do not materially alter the likelihood of

the order being filled. By example, when a stock is quoted 50.0-50.2,

then an order change from 45 to 46 does not really change the likelihood

of execution.

Most clients are able to easily improve their order management by

introducing 'no-waste' logic into their systems. We kindly request you

examine your order management logic to reduce unnecessary orders to the

recommended levels."

"For additional information, including suggestions as to how to reduce

waste in your order management behavior, please type KB1343 ( IB../1343

) into the search engine located on the IB home page or contact our API

services group at api@interactivebrok ers.com

Interactive Brokers Customer Service"

read more:

https://groups.io/g/twsapi/topic/4044829

ZenFire seems a very attractive choice from a programmer's perspective

[Q] Some IB API timing requirements

[A1] by Bill Pippin (from this thread)

The api not only imposes timing requirements on requests in

general (20 msec for the IB tws, or ~6 msec for the gateway); it

also routinely rejects child orders that follow too closely on the

heels of the parent (folklore says 300 msec here); same symbol

history queries at the rate of more than 6 each two seconds; and

history queries in general, at the rate of more than 60 per ten

minutes. Worse yet, the history query pacing requirements seem to

be enforced by the upstream, so there are possible races caused by

network jitter to contend with.

[A2] by Angel@IB support by phone:

. The limits I referred to was the reqContractDetails( ) limit, please see the details below: 

Note: IB implements a limitation when invoking reqContractDetails( ) that will buffer the contractDetails( ) call back for any subsequent requests from 5 –  60 sec. To avoid the pacing limitation (up to 8 requests) it is essential that the FULL expiration is provided within the m_expiry parameter in the format YYYYMMDD as well as the Strike, Right and Multiplier. Otherwise if you request contract details providing the exact parameters outlined above it would be buffered by 5 sec.

For Examples:

If Expiry  = Blank or NULL

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to null or empty string, the delay is 1 minute.

If Expiry  = YYYY (e.g.  2014) 

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “2014” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year only, the delay is 1 minute.

If Expiry  = YYYYMM (e.g. 201402) 

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “201402” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, if the expiry is assigned to year and month format (YYYYMM), the delay is 5 seconds.

If Expiry  = YYYYMMDD (e.g. 20140221) 

Symbol = “IBKR”;

secType = “OPT”;

Expiry = “20140221” ;

Exchange = “SMART”;

Currency = “USD”;

Above example, please note, that this ONLY works with front month option contracts, if the expiry is assigned to year with month plus the day (YYYYMMDD), then there is NO delay.

With regards to the historical data pacing limits please refer to the following URL: 

https://www.interactivebrokers.com/en/software/api/apiguide/tables/historical_data_limitations.htm 

Regards,

-Angel 

IB API Support 

Dmitry copy-n-pasted the content of the above URL “historical data limitations” for completeness of this codument:

——————————————————– (quote)—————————————–

Historical Data Limitations

Historical data requests are subject to the following limitations:

For All Securities

  • Tick-by-tick data is not relayed back through any API technology, we only provide the raw values used to calculate the time bars.
  • Studies and Indicators are not relayed back through any API technology.
  • Intra-day bar sizes are relayed back in Local Time Zone, daily bar sizes and greater are relayed back in Exchange Time Zone.
  • For daily bar sizes and greater, the date value will only return in format "yyyymmdd", formatDate = 2 are only supported for intra-day bars.

For Stocks, CMDTY, ETFs, Forex, Indices and CFDs

  • Historical data requests that use a bar size of "30 secs" or less can only go back six months.
  • Historical data requests can go back one full calendar year or more, depending on the number of concurrent real-time market data lines:

Number of Market Data Lines

Historical Data

Request Limit

499 or less

One year

500 – 749

Two years

750 – 999

Three years

1000 -1249

Four years

1250 – 1499

Five years

 

Market data lines can be increased based on monthly commission amounts, amount of equity and Quote Booster subscriptions. A "market data line" refers to a quote line in TWS and to each reqMktData() and reqRealTimeBars() method invoked in the API.

  • For more information on how market data is affected by commissions and equity, expand the "How Market Data is Calculated" section of the Market Data Display page on our website.
  • Quote Boosters are counted on top of your current real-time market data line limits. For more information on how market data is manually affected by quote booster subscriptions, see the Quote Booster section of the Market Data Display page. You subscribe to Quote Boosters on the Market Data Subscriptions page in Account Management.

The following table lists the monthly commission required, equity required, and/or quote booster packs required to increase the total years of historical data:

Total Years of

Historical Data

Com x Data = Mthly Com

Mthly Commission Required

Equity x Data = Mthly Equity Data

Mthly Equity Required

Data – Default = Data Needed

|Data Needed/Per Booster| = Total Booster

One year

Less than USD 4000

Less Then USD 5,000,000

3 Booster packs or less

Two years

USD 8.0 x 500 = USD 4,000

USD 10,000 x 500 = USD 5,000,000

500 – 100 = 400

|400 / 100| = 4

4 Booster Packs

Three years

USD 8.0 x 750 = USD 6,000

USD 10,000 x 750 = USD 7,500,000

750 – 100 = 650

|650 / 100| = 7

7 Booster Packs

Four years

USD 8.0 x 1000 = USD 8,000

USD 10,000 x 1000 = USD 10,000,000

1000 – 100 = 900

|900 / 100| = 9

9 Booster Packs

Five years

USD 8.0 x 1250 = USD 10,000

USD 10,000 x 1250 = USD 12,500,000

N/A (Max Quote Booster Packs is limited to 10)

Note:  Increasing how far back historical data is available will have no effect on the pacing limitations. Pacing limitations are hard coded on IB’s Server backend and as such not user adjustable. Please see the below Pacing Violation section for more details.

For Futures, SSFs and Metals

  • Historical data requests are available for expired Futures, 2 years from current expiration.
  • Historical data requests are typically available for up to 6 months per expiration.
  • Historical data requests require the expiration to be specified, at this time continuous contract is not supported.

For Options, FOPS, Warrants and Structure Products

  • Historical data requests are only available for current expirations.
  • No End of Day (EOD) data are available, the valid bar sizes are from “1 secs” to “8 hours”
  • Historical data requests are typically available for up to 2calendar months per expiration.

For Bonds and Funds

  • Historical data requests are not available through the API, only real time data can be requested.

Note:  For more information on requesting real time data see Using the Tickers Page in the DDE for Excel chapter, reqMktDataEx() in the ActiveX chapter, reqMktData() in the C++ chapter, reqMktData() in the Java chapter, reqMktData() in the C# chapter and Using the Tickers Page in the ActiveX for Excel chapter.

For Real Time Bars

When requesting Real Time Bars this is a query for streaming Historical Data, the data is relayed back from the same servers that provide Historical Data. As such when initiating the request it is subjected to the same pacing limitations outlined below in the Pacing Violation section. Also since the Real Time Bars requests are being provided as streaming, it will count towards the total number of Market Data Lines outlined in the Market Data Display page on our Website.

Note:  Real time bars not available for DDE. For more information about real time bar requests, see reqRealTimeBarsEX() in the ActiveX chapter,reqRealTimeBars() in the C++ chapter, reqRealTimeBars() in the Java chapter, reqRealTimeBars() in the C# chapter and Real Time Bars in the ActiveX for Excel chapter.

[Q] What is Pacing Violations?

All of the API technologies support historical data requests. However, requesting the same historical data in a short period of time can cause extra load on the backend and subsequently cause pacing violations. The error code and message that indicates a pacing violation is:

162 – Historical Market Data Service error message: Historical data request pacing violation

The following conditions can cause a pacing violation:

  • Making identical historical data requests within 15 seconds;
  • Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds.

Also, observe the following limitation when requesting historical data:

  • Do not make more than 60 historical data requests in any ten-minute period.
  • If the whatToShow parameter in reqHistoricalData() is set to BID_ASK, then this counts as two requests and we will call BID and ASK historical data separately.

Note:  For more information about historical data requests, see Viewing Historical Data in the DDE for Excel chapter,reqHistoricalDataEx() in the ActiveX chapter, reqHistoricalData() in the C++ chapter, reqHistoricalData() in the Java chapter,reqHistoricalData() in the C# chapter and Viewing Historical Data in the ActiveX for Excel chapter.

Valid What To Show Values

The following table lists the valid whatToShow values based on the corresponding products.

*Only applies for CFD Indices. For CFD stock, you must specify the underlying stock.

Valid Duration and Bar Size Settings for Historical Data Requests

The following table lists the valid duration andbar size settings for API historical data requests. Please note that these are only guidelines but if you attempt to exceed them then each request may count as more than one and can cause a pacing violation as outlined above.

——————————————————– (end of quote)—————————————–

[Q] Can I be long and short on the same underlying same time?

[A] Short answer – no. If you enter long and then short same instrument you’d be flat.

But there’s other size of this question.

[Q] Any historical-data aggregating programs?

[A] by Peter Gum

Let me suggest looking at some of the historical-data aggregating programs.

Jtwsdump and ibcollector come to mind (google those names).

-Pete

[Q] How to diagnose an issue?

[A] by Kurt

Two basic things to always check when diagnosing "beginner" problems:

* try the corresponding functionality in TWS

(look at P&L columns, maybe both in trading screen & acct window)

* try doing the same thing (reqAccountUpdates) in the sample API app

Doing those two things sheds light on the majority of problems, in my

experience. It eliminates any possible issues with your own code, and makes

it straightforward when you contact IB for help.

Maybe also try it on the demo account, and paper account, and see if you get

any different results.

But if it behaves just as badly with the sample app, I'd say you are seeing

a problem I have never seen nor remember hearing anyone else here talk

about, and at that point it is probably time to talk to IB. No reason this

should work as badly as you are describing and if you don't address (going

instead for a workaround) it you may see problems elsewhere.

-Kurt

[Q] Why should NOT I use DDE?

[A] by Abhijit Dey

I can attest to this DDE instability problem. I like the quick modeling

capabilities of Excel and like to fire off some orders from there.

However, every time I open a spreadsheet that has say 50+ tickers

subscribed, chances are good Excel is going to blow up. Most times the

machine gets so messed up, I need a reboot (before Excel DDE would work

again). I have moved from excel 97 format to the new version (office

2007), but that has not helped at all. Same experience in both Vista 64

and xp 32.

Truly a sad state of affairs…

[Q] Is demo account API behavior differ from the behavior of real account?

[A] by Kurt

That was my experience a year or two ago.

The demo account is useless for quite a few things. Even historical bars

were useless in relation to testing my code because the anomalous behavior

was so bazaar, including strange kinds of resets of the data that occurred

at various particular times of day and/or week. At certain times (including

I think after the friday night reset) there would be no bars, i.e. the

number of bars returned would always be zero, for everything.

I actually wrote code to create crappy historical bars, so that I could test

my code over the weekend (when it used to be the case that only the demo was

running). My 120 lines of code worked better for me than the demo login,

for historical data. I may have also simulated realtime bars, I forget.

So I stopped using the demo account except when I have some really strange

reason to actually need it. I needed it earlier to get market data ticks

coming in to exercise the conditions for a certain bug.

-Kurt

[Q] What are "EX" methods?

[A] by Kurt

’I think the Ex methods are an ActiveX thing. They are at least described in

the release notes if not in the ActiveX section of the API doc.

If you are using Java or C++ the Ex methods are not a concern.

-Kurt

[Q] custom indicators?

[A] by baskosi

This is a reply I got from IB on 2009/11/30:

Unfortunately, custom indicator is no longer supported. Additionally, we do not provide sample code for customer indicator.

In the API environment, we only provide raw data, it is up to user to implement an indicator to associate the raw data. Unfortunately, we do not have any sample of custom indicator.

IB API Support

[Q] Any other tips/hints for beginners please?

[A] by Kurt

Generally I request contract details first on everything, and save the results. The

resulting contract then gets used for subsequent requests.

by Kurt

(few months later…)

So the only thing I can suggest is that placeOrder is more fussy about the

Contract object than requests for contract details or market data. A

general rule of thumb is to request contract details to obtain the contract

id. Then use that contract id with all other requests including placeOrder.

If you specify contract id you do not have to specify any other fields, and

you eliminate several sources of possible error.

The only difference between what your code does and mine is that you are

setting localSymbol, whereas I leave it set to the null string. Local

symbol is needed to distinguish individual contracts with the same

underlying if you do not specify the expiry. Other people recently have

reported that code stopped working that used to work, when they specify

fields that are not required. My impression is that non-required fields

used to be ignored in more contexts, and now IB is reporting errors for

these.

–Kurt

[A]

"As I said once before, it will probably be useful to stay subscribed to account

updates on an ongoing basis." – Kurt Bigler <kkb@…>

I second that suggestion. More than once I've had IB add unexpected positions

intraday (spin-offs, busted trades, etc).

–Frank Bell

[A] interesting technique to make orders fast (by FutureScalper):

I should say that my standard way of striking the market is NOT to

place a limit order where I want to buy. Instead, my system checks

various conditions multiple times per second in order to determine

when and at what price to strike. Then it moves a pre-loaded limit

order, which has been sitting outside the market, to the strike price.

That pre-loaded limit keeps itself outside the market, by modifying

its price as economically as possible, but it does have to move to

stay sufficiently outside the market until it is needed. It might

stay there for several minutes prior to striking, and may never have

to strike, and so be cancelled as well, of course, if the conditions

never appear. I call this method the "Virtual Limit Order".

This technique avoids intial order placement delays and margin

checking overheads. By moving the pre-loaded order to a different

price, the strike time is faster than if a fresh order were to be

submitted when needed. Anyway, so that makes me an "excessive"

modifier although, again, that's pretty modest when compared with the

really abusive trade-bots out there. 🙂

Sitting about 45 msecs ping time round trip from the exchange, my

order strike latencies are in the region of 60-80 msecs using this

technique. Of course the order has already struck and probably been

filled by the time I receive the ack that it's been moved. So

immediately after that, usually, I get the ack for the actual fill.

[A] by Kurt (good one:)))

IB probably added some bugs as well, since IB's bug-to-feature ratio is

always pretty high. You are the tester. That's how you pay when the

commissions are so low.

-Kurt

[A] by Kurt

the EClientSocket source is in effect the documentation for the API socket protocol. It is

freely and publicly available.

[A] Any time you're using sleep() to avoid a race, you've got a design flaw!

[A]

The problem with off-the-run futures contracts (and

any option, or any thinly traded equity for that matter)

is that the "last" price is often incredibly stale, and the

only true indicator of the market is, well, the market

(the bid and ask).


IB Gateway app related

[Q] Installing IB Gateway and TWS API for linux?

[A] by Rich Holowczak (a.k.a. “Prof. H.”)

Folks:

For anyone who is *just* getting started with IB API programming under Linux, I've written up a step-by-step guide to setting things up under Linux. I've included the basic steps for compiling the POSIX C++ test client as well. Looking forward to adding on to this and for your suggestions.

Installing Interactive Brokers IB Gateway and TWS API for Linux

[Q] Trader Workstation (TWS) vs IB Gateway?

The Gateway lacks the GUI of the TWS, but provides the same API.

[A1] spotted on stackoverflow

IB writes about the performance:

The IB Gateway provides a low-resource alternative to TWS for connecting to the IB trading system via the API. The gateway uses approximately 40% fewer system resources than TWS. However, the gateway is GUI-less, which means that you cannot view the API activity as you can when running TWS.

[A2] by IB’s tech support

From the API point of view both platforms are identical. The only difference between them is that the TWS has a much richer interface through which you can place orders, view charts, etc. The IB Gateway is just a proxy with a simple interface aimed at providing the most basic features in order to connect via the API.

More important though, the TWS requires to be restarted at least every 24 hours whereas the IB Gateway can run indefinitely.

[Q] Any performance difference between IB Gateway vs TWS app APIs?

[A] by Jian Ren

The gateway has higher message limit than TWS, 200 vs 50. If you scalp many

stuffs at the same time, 50 might not be enough.

The gateway is a bridge between TWSAPI and FIX, so anything written for the

TWSAPI could work without any change(could anyone confirm?) after switching.

For what you said, you might only need a FIX connection, not sure whether IB

provides to retail customers.

Jian

but Kurt thinks “gateway has higher message limit” is not true. Quote:

Jian Ren,

The description of the Gateway login is certainly confusing.

Are you sure that if you choose FIX login from IB Gateway, and use the

Market Data connection in that context that the higher message limit will

apply to the TWSAPI portion of the connection?

On the surface that contradicts the info on this page which ascribes a lower

message limit to the TWSAPI:

http://individuals.interactivebrokers.com/en/pagemap/pagemap_APISolutions.ph

p

It is true that the FIX CTCI page accessed from there, also states

"The IB Gateway can also be used to deliver market data to FIX CTCI

customers, using the same protocols as the IB API"

(and note the word "also"). But that does not give me the impression that

the IB Gateway provides two entirely distinct modes for using the TWSAPI,

one of which has the higher (FIX) message limit. "The IB Gateway Software"

documentation also seems to suggest that FIX customers are merely permitted

an ordinary TWSAPI login "on the side" so to speak, and I would think that

the usual TWSAPI message limits would apply. The fact that the TWSAPI

portion of the connection requires a separate login also suggests this.

Also it does not appear to me that the IB Gateway is a bridge between TWSAPI

and FIX, specifically because FIX offers only a subset of TWSAPI

functionality (as per the chart from the url referenced above), but again

also because of the two distinct logins required.

-Kurt

[Q] Gateway Crash on Startup – Linux

Wondering if anyone else has seen this error when starting TWS or GW on a Linux install and if you did how it was resolved.

Glib-GIO-ERROR Settings schema `org.gnome.system.proxy' is not installed

by don_brockhage don.brockhage@gmail.com via yahoogroups.com

[A] Yes, I had to install gsettings-desktop-schemas.

Leon

by Leon Baum leonbaum2@gmail.com via yahoogroups.com  

[Q] list of the error 321 "message is not supported"

[A] by Kurt Bigler

I'm putting together here a list of the error 321 "message is not supported"

errors people have encountered so far when using the new gateway, plus

results from additional testing I have done.

I'm cataloging these according to the two-letter code which is part of the

error 321 text as in Error validating request:-'uc'

2-char ———————cause———————————–

code previous reports my C++ app TestJavaClient button

'uc' SET_SERVER_LOGLEVEL

'tc' setServerLogLevel Server Logging

'pc' (unknown)

'ec' reqRealTimeBars

'dc' reqRealTimeBars Req Real Time Bars

'sb' cancelRealTimeBars Cancel Real Time Bars

'lc' requestFA Financial Advisor

'oc' Req News Bulletins

'nc' Req Accounts

Curiously in a couple places it looks like two different codes are getting

used for the same general type of request. I haven't investigated this

further.

And note that while Req Accounts is not supported, the "list of managed

accounts" is still sent automatically after connect, which is all you really

need.

This is all on the Mac, using TestJavaClient 9.62, and my C++ app which is

roughly based on 9.60.

-Kurt

[Q] autologin script for IB Gateway?

[A] by -Castedo

You can also use IB Gateway which is essentially a striped down version of TWS. Unlike TWS, IB Gateway will try to maintain a connection to IB for much longer than 24 hours. There is no guarantee that IB Gateway will keep the connection up for more than a week but this does happen.

So rather than trying to do autologin and store your password on your server, you can just run IB Gateway on your server, use the IB secure login system (using the IB security device), NOT store your IB password on your server and just re-authenticated manually as needed.

If you want to run on a Linux server and re-authenticated via a HTML UI or Web API and not a GUI desktop then consider Brokertron Gateway for IB:

http://www.brokertron.com/gateway/

-Castedo (developer of Brokertron Gateway for IB)

[Q] How to automatically accept client connections to TWS and/or Gateway from any remote IP address (not known in advance)?

[A] by Dmitry

Other simple option to accept remote clients from unknown (in advance) IP's would be to use passwordless ssh tunnel.

[enable ssh on your Gateway/TWS host]

if your Gateway/TWS is on OS-X

    google: mac os x enable ssh daemon

    make sure you can ssh into your box, simplistic case would be to try to ssh to localhost (from the box itself:)

    ssh youusername@localhost

if you don’t have any simple means to run ssh daemon on you Gateway/TWS host, then

you can use other machine from you LAN to play the role of “man in the middle”.

Your remote clients will establish ssh tunnels to that intermediary box, which will

in turn try to connect to Gateway/TWS and this time all the connections

will come from static IP address which you will configure to be trusted once only.

[open port forward on you firewal]

if you behind your NAT and all you LAN addresses are like 192.168.x.x or 10.0.x.x or from other private address IP range

(not directly exposed to the internet) then you’ll need to create port forward rule on your firewall so it would listen on say

port 1022 and forward it you your local machine port 22 (ssh)

Look at the image: http://factorial.hu/system/files/images/local_0.png

it shows the basic ssh tunnel along with ssh command to establish this beauty!

[no need for 3rd box for tunnel to work]

I’m suggesting to use 2 computers instead of 3 on the image above.

In this case your ssh daemon listening on port 22 and waiting for

remote clients to connect wold resides on the same box as your Gateway/TWS app.

Imagine joeuser@home and remote.service.com are on the same box (which runs Gateway/TWS).

Then all incoming connections to the Gateway/TWS will come from 127.0.0.1

[let us run the tunnel]

On remote client side you’d initiate the tunnel like this:

ssh -p 1022 -L 7496:127.0.0.1:7496 dmitry@your_router_host_or_ip.com

“-p 1022”   is the port you accept incoming ssh connections on you router (and forward to GW/TWS host)

“-L 7496:”  would create local port 7496 on your remote client

            so any app you run to connect to GW/TWS will use “127.0.0.1:7496” as if

            it all runs locally on the same host as GW/TWS itself!

            This can be any port (better >1024 and not used by other apps:)

“127.0.0.1:7496”  this part describes where all the packets will be directed to after they

                  pop out of the other (remote for your client app) end of the tunnel. As you

                  can see they will go to localhost running GW/TWS std port.

[let us test the tunnel]

On remote client run simple telnet command:

telnet 127.0.0.1 7496

    Trying 127.0.0.1…

    Connected to 127.0.0.1.

    Escape character is '^]'.

The output highlighted in green suggests we have established connection (somewhere:)

Now watch your TWS application before and after you run the above telnet command – you'll

noticed it has accepted remote client (since for the GW/TWS client came from 127.0.0.1) and you

can see new "API" tab opened (if you use TWS, I'm not sure how Gateway would react on

connected client).

[make it passwordless]

Now google "passwordless ssh" and generate a key pair for you ssh clients and here you go!

No more questions "Accept incoming connection attempt from …." asked! And your client can come

securely from all over the world, all communication is encrypted.

To get your router IP (even if it dynamically changes from time to time by your ISP) you can use DnyDNS for $25/months, or google for "dyndns alternative"

[enjoy!]

this is the most important and often ignored item on each and every list of "things todo"

🙂


Some Forex notes

[Q] FXCONV support in API?

I searched for FXCONV (for Forex) in API documentation but see no mention of it. Is it not supported by API at all and only option is IDEALPRO?

IDEALPRO is bad because it is slightly higher commision fees AND it makes a useless / confusing virtual FX position.

[A] by Josh View/Reply Online (#46636) 

Added on 27-Feb-2021

Yes it’s supported, you just have to set the exchange field that way (instead of ‘IDEALPRO’). By the the way, you can also turn off the virtual positions reporting through the settings if you don’t want to receive them.

yes 'FXCONV' is only for placing orders, just like in TWS. There is a setting in TWS at Global Configuration -> API -> Settings -> 'Include Virtual FX positions when sending portfolio' that can be unchecked to prevent virtual positions from being returned.

[Q] IDEALPRO LMT orders at the top of the order book never executed

Dear All,

I am wondering if anyone of you has experience trading on IDEALPRO.

I coded a trading system which places BUY and SELL LMT orders at the top of the order book using reqMktDepthEx. Limit prices have been continuously updated according to order book. The issue is that the system never makes any trade although my orders are sitting at the top of order book all day long. I have tried EUR.ILS and EUR.JPY with the same result.

Do you have any suggestion how to solve this issue?

Regards

Martin

[A] found in this thread

Added on 09-Sep-2015

Dear All,

Thanks for help and interesting discussion. Finally, I have found someone in IB who explained the situation to me.

Please see quotation below:

 

If IB client A becomes the best bid (better than bank liquidity providers) and IB client B wants to sell, they can execute against your bid.

Only IB's clients can actively hit IB client A's bid. This is because we do not expose our internal orders to outside third parties.  

USD.ILS is a very illiquid currency, so there may be no interest in IB client A's bid for some time. In the case of the EUR.USD this would most likely be different, since this is a very liquid currency. If no one within IB hits IB client A's bid, the bid remains there until it's matched against another IB client order or it meets the bank liquidity providers offer, in which case it will be executed.  

I hope this provides more insight into how our platform works.

 

The point is that only IB's client can hit my order. There are probably very few IB's clients trading EUR.ILS or EUR.HUF. That is a pity that IB does not expose the orders to outside. In the situation, when the entire FX market creates only IB's clients there is almost no liquidity on many FX pairs which makes IDEALPRO unusable for such kind of operations.

According to IBI, DEALPRO covers 78% share of the global FX market and 14 of the largest FX dealing banks participate. Unfortunately, this statement is very misleading since only a very small share of the global FX market can see my orders!!!!

My model is based on tick data coming from Bloomberg and Reuters which probably show much larger liquidity on EUR.ILS (as well as on other FX pairs) than is provided by IB's clients. I have tested several data sets coming from different providers (overall coverage of my data sets is more than 90% of the global FX market).

 

I have found IB quite a good broker so far but this misleading statement regarding FX global market coverage cannot be accepted.

 

Do you know, is it possible to get IB's tick data so I can compare it to my data sets?

Regards

Martin

[Q] Please check my Forex math!

I have been staring at this long enough to totally confuse myself about a couple of points, and I'd really appreciate it if someone here could straighten me out.

Consider the following:

public static Contract getEurUsdForex()

{

  Contract contract = new Contract();

  contract.Symbol = "EUR";

  contract.SecType = "CASH";

  contract.Currency = "USD";

  contract.Exchange = "IDEALPRO";

  return contract;

}

 

Order order = new Order();

order.OrderType = "MKT";

order.Account = AccountNumString;

order.TotalQuantity = 80000

order.Action = “BUY”;

ibConn.ClientSocket.placeOrder(nextOrderId++, getEurUsdForex(), order);

 

If I have a starting position (in the FXTrader window under EUR.USD) of -40K

Question #1:

If I execute this, and the order is filled at 1.3900 (and my new position is +40K)  ,  Have I bought 80K Euros at $1.39 each, or have I bought $80,000 worth of Euros at $1.39 each?

Question #2:

If I then change the Action to “SELL” and the order is filled at 1.3901 , (and my new position is -40K) do I have a net gain [ON THIS TRADE] of 0.0001 * 80K Euros at $1.3901 each =  8 Euros = $11.1208?

Would the commission on this trade be 0.00002 * 80K * $1.3901 = $2.224 ?

[A] by vanx23@outlook.com

Answer #1:

You will have bought 80,000 Euro units. For a total worth of 111,200 USD.

Answer #2:

You will have a commission fee on your BUY, as well as on your SELL, transactions.

BUY 80,000 @ 1.39 = 111,200 USD * 0.00002 = 2.224

SELL 80,000 @ 1.3901 = 111,208 USD * 0.00002 = 2.22416

Total commission: 4.448 USD

Gain/Loss = 8 USD

Net profit: 3.55 USD

Note: I don't buy currencies through the API. I use TWS for that purpose.

[Q] according to the documentation m_avgPrice should include commissions, but it doesn’t!!

[A] by rwk (from this thread)

I don't trade forex, so I could be wrong, but it was my understanding that there is no commission when trading forex at IB.  The broker's compensation is gained through the bid-ask spread, so you never actually see it.

[rwk]

[Q] Can't get some Forex historical data?

[A] by Kurt

IB has its concept (or maybe it has meaning in a larger context) of native

currency pairs. Only the native pairs are supported and not the reversed

pairs. The native pairs are listed on the IB website somewhere, but here

are the ones I use along with their conId value (which you don't need to use

but was convenient for my copy/paste).

USD.CHF 12087820

EUR.USD 12087792

NZD.USD 39453441

USD.NOK 37971206

USD.JPY 15016059

USD.SEK 37893486

GBP.USD 12087797

AUD.USD 14433401

USD.CAD 15016062

USD.SGD 37928772

USD.MXN 35045199

USD.HKD 12345777

KRW.USD 27717934

-Kurt

[Q] Why I don’t get TRADES from “real-time” FX ticker?

[A] by Kurt (from this thread)

Sorry I don't have helpful code laying around. Other people have used

different approaches for specifying FX contracts, but the method I use for

example with EUR.USD is to specify the EUR as the symbol, USD as the

currency, CASH as the security type, and IDEALPRO as the exchange.

I believe all FX pairs will only provide historical data in one direction,

i.e. you can chart USD.CHF but not CHF.USD, even though tick data and

execution is provided for both. Also you can only chart IDEALPRO, never

IDEAL.

Also with FX you can not get data for TRADES, only BID, ASK, BID_ASK, and

MIDPOINT.

Finally some FX pairs have no historical data at all (as far as I can tell)

such as HKD.USD (or USD.HKD). HKD.USD is IDEAL-only, and USD.HKD is

IDEALPRO but can not be charted.

-Kurt

[Q] Tracking Forex positions?

[A] by Kurt

  • I've just started looking at the values updatePortfolio() returns. I
  • /thought/ that 'positions' would return a quantity but the numbers I'm
  • getting are much too large to be that.
  • E.g. A forex position for AUD.USD quantity=154171 BOT has been filled
  • at 00:09:03 (according to the TWS 'Trades' page.)
  • The first updatePortfolio() event with m_localSymbol=AUD.USD arrived at
  • 00:14:17 and reports
  • position=-13440588
  • What does this position value mean? (For the record, IB's documentation
  • says "This integer indicates the position on the contract. If position
  • is 0, it means the position has just cleared.")
  • FYI this result is from paper trading, IB api version 9.4.
  • thanks
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Jun 5, 2010
  • View Source
  • On 6/4/10 9:41 AM, "Barnet Wagman" <b.wagman@…> wrote:
  • > I've just started looking at the values updatePortfolio() returns. I
  • > /thought/ that 'positions' would return a quantity but the numbers I'm
  • > getting are much too large to be that.
  • >
  • > E.g. A forex position for AUD.USD quantity=154171 BOT has been filled
  • > at 00:09:03 (according to the TWS 'Trades' page.)
  • >
  • > The first updatePortfolio() event with m_localSymbol=AUD.USD arrived at
  • > 00:14:17 and reports
  • >
  • > position=-13440588
  • >
  • > What does this position value mean? (For the record, IB's documentation
  • > says "This integer indicates the position on the contract. If position
  • > is 0, it means the position has just cleared.")
  • >
  • > FYI this result is from paper trading, IB api version 9.4.
  • I do believe others may have reported flukes with positions in paper trader,
  • but I can not recall details. I don't use paper trader for anything
  • serious. 😉
  • But TWS also has serious flukes with their notions of positions in the case
  • of FX. You don't *get* positions with FX. You get "virtual positions" or
  • something. You get the net position you have taken in relation to the
  • *particular* currency PAIR in question, and after years and years you may
  • never be able to get that to zero out, depending on exactly what sequence of
  • currency pairs you have traded during that time.
  • Personally I ignore the updatePortfolio results and use the
  • updateAccountValue results instead. I look at the CashBalance value for
  • each currency (except BASE) to determine net position in that *currency*
  • (not PAIR). I also use the ExchangeRate value to determine an effective
  • displayable market value for that currency in relation to my base, and
  • compute a position value in my base.
  • It works well for display. For trading you probably want to model your own
  • position value based on order status and execution reports, and perhaps
  • reconcile it dynamically against the somewhat delayed updateAccountValue
  • information whenever it comes in, since occasionally something is missing
  • from the data stream that you may need to have a correct model of your
  • positions.
  • I personally would not bother with using the virtual FX positions through
  • the API. When trading in TWS itself I am kind of stuck with that, and make
  • do, but it gets worse as the years go by.
  • -Kurt
  • Reply
  • Kurt Bigler
  • Jun 6, 2010
  • View Source
  • To answer your original question…
  • When I first looked at the large negative number I figured you had just run
  • into a bug. It is conceivable that your "virtual" FX position on that pair
  • might be negative 13 million in paper trader, if your net activity on
  • AUD.USD was to short that much of that pair. But if the single trade you
  • named was the only trade on AUD.USD you have done, then that should not be
  • the case and I go back to the idea that it is a bug. Either that or
  • something went wrong with API message parsing or the code you are using to
  • log values–but in java that should not come up (as it could in C/C++ with
  • weak compiler warnings in a printf).
  • Interesting you are the 2nd person (recently) to mention using a currency
  • pair name as the local symbol, i.e. rather than specifying "USD" in the
  • currency field. Even more interesting that it shows up via updatePortfolio
  • in this form, which I would not have expected. Someone else in the last
  • couple weeks indicated behavior changed recently in relation to this
  • "feature" when specifying an FX Contract, but I don't know whether it will
  • affect you on 9.4.
  • -Kurt
  • Show message history
  • Reply
  • Barnet Wagman
  • Jun 6, 2010
  • View Source
  • Thanks for the information on updatePortfilio – I'll definitely avoid
  • it. I'm currently maintaining my own balances based on order execution,
  • as you suggest. I'm glad to hear that the updateAccountValue messages
  • are useful, at least as a check on my own numbers.
  • Re the 'local symboll':
  • I use the 'local symbol' to identify forex transactions because they
  • reflect the actual 'contract' that an IB forex order represents.
  • E.g. an order for EUR.USD is specified as quantity of EUR. In
  • IdealPro, there is no USD.EUR 'contract', so to buy USD with EUR, it
  • necessary to sell EUR.USD. As I understand it, in IdealPro, there is
  • only one contract (local symbol) for each pair of currencies, at least
  • at present (from what I've read, at one time there were more pairs).
  • Using the local symbol makes things a little simpler.
  • However, when I construct a Contract object, I set ALL the currency fields:
  • contract.m_currency
  • contract.m_symbol
  • contract.m_localSymbol
  • I don't know if setting all three is necessary, but it does appear to work.
  • bw
  • Kurt Bigler wrote:
  • >
  • >
  • > To answer your original question…
  • >
  • > When I first looked at the large negative number I figured you had
  • > just run
  • > into a bug. It is conceivable that your "virtual" FX position on that pair
  • > might be negative 13 million in paper trader, if your net activity on
  • > AUD.USD was to short that much of that pair. But if the single trade you
  • > named was the only trade on AUD.USD you have done, then that should not be
  • > the case and I go back to the idea that it is a bug. Either that or
  • > something went wrong with API message parsing or the code you are using to
  • > log values–but in java that should not come up (as it could in C/C++ with
  • > weak compiler warnings in a printf).
  • >
  • > Interesting you are the 2nd person (recently) to mention using a currency
  • > pair name as the local symbol, i.e. rather than specifying "USD" in the
  • > currency field. Even more interesting that it shows up via updatePortfolio
  • > in this form, which I would not have expected. Someone else in the last
  • > couple weeks indicated behavior changed recently in relation to this
  • > "feature" when specifying an FX Contract, but I don't know whether it will
  • > affect you on 9.4.
  • >
  • > -Kurt
  • >
  • > On 6/4/10 9:41 AM, "Barnet Wagman" <b.wagman@… 
  • > <mailto:b.wagman%40comcast.net>> wrote:
  • >
  • > > I've just started looking at the values updatePortfolio() returns. I
  • > > /thought/ that 'positions' would return a quantity but the numbers I'm
  • > > getting are much too large to be that.
  • > >
  • > > E.g. A forex position for AUD.USD quantity=154171 BOT has been filled
  • > > at 00:09:03 (according to the TWS 'Trades' page.)
  • > >
  • > > The first updatePortfolio() event with m_localSymbol=AUD.USD arrived at
  • > > 00:14:17 and reports
  • > >
  • > > position=-13440588
  • > >
  • > > What does this position value mean? (For the record, IB's documentation
  • > > says "This integer indicates the position on the contract. If position
  • > > is 0, it means the position has just cleared.")
  • > >
  • > > FYI this result is from paper trading, IB api version 9.4.
  • > >
  • > > thanks
  • >
  • >
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Message 5 of 5 , Jun 6, 2010
  • View Source
  • On 6/6/10 2:42 PM, "Barnet Wagman" <b.wagman@…> wrote:
  • > Thanks for the information on updatePortfilio – I'll definitely avoid
  • > it.
  • Just for FX!
  • Also you can use live market data to get faster updates on market value than
  • you'd get via updatePortfilio–unless you don't have market data in which
  • case updatePortfilio will at least give it to you for positions you hold.
  • > I'm currently maintaining my own balances based on order execution,
  • > as you suggest. I'm glad to hear that the updateAccountValue messages
  • > are useful, at least as a check on my own numbers.
  • >
  • > Re the 'local symboll':
  • > I use the 'local symbol' to identify forex transactions because they
  • > reflect the actual 'contract' that an IB forex order represents.
  • >
  • > E.g. an order for EUR.USD is specified as quantity of EUR. In
  • > IdealPro, there is no USD.EUR 'contract', so to buy USD with EUR, it
  • > necessary to sell EUR.USD.
  • Yes, but I see that as selling EUR, in USD currency. That is symbol EUR,
  • currency USD, no local symbol. I never used the local symbol field in
  • Contract (for FX). But in fact I never did currency trades via API, only
  • via TWS, so maybe I'm missing something. But even TWS, if you don't use the
  • "FX Trader" window seems to think in the terms I just described. You first
  • type EUR, then choose currency USD from the list (and IDEALPRO or IDEAL).
  • I do get contract info and market data via API, and I do it as I described
  • (no local symbol). It is really curious to me people are using local symbol
  • and I wonder if this is just for "historical" reasons because IB didn't
  • implement things consistently "in the early days".
  • > As I understand it, in IdealPro, there is
  • > only one contract (local symbol) for each pair of currencies, at least
  • > at present (from what I've read, at one time there were more pairs).
  • I guess I've seen that sometimes. But I do believe I've traded some in both
  • directions, much to my later chagrin since I believe it complicated my
  • "virtual" positions scenario in TWS. Certainly market data appears for
  • currency pairs stated both directions. Often the liquidity does not seem
  • the same and the markets even appear to be distinct, like no one is doing
  • arbitrage, which baffles me, but may be the nature of "spot", and the spread
  • and commissions may leave room for the prices to drift if no one can make
  • money equalizing it. I'd love to hear opinions/explanations about this.
  • I'm not a big currency trader trader–I use it for longish-term hedging with
  • only occasional trades, and never in IDEALPRO quantities. I have to use
  • IDEALPRO to get historical data because there is none for IDEAL.
  • > Using the local symbol makes things a little simpler.
  • I'm not clear how it is simpler. It seems more complex in being less like
  • non-FX. So I'm wondering if you can clarify whether you get some additional
  • functionality by using local symbol.
  • > However, when I construct a Contract object, I set ALL the currency fields:
  • >
  • > contract.m_currency
  • > contract.m_symbol
  • > contract.m_localSymbol
  • Well that might seem to negate what you said earlier that you use local
  • symbol to simplify something. I'm not trying to argue, just understand.
  • Local symbol only *exists* in the Contract object, and if you set it, but it
  • is ignored (see below) then in that case it would not account for any of the
  • functionality you experience, thus not "simpler".
  • I was trying to imagine how your approach might somehow make currency pairs
  • easier to navigate, when perhaps only one direction has liquidity.
  • > I don't know if setting all three is necessary, but it does appear to work.
  • It may be localSymbol used to be ignored for currency contracts, and perhaps
  • still is via 9.4 api, but is no longer ignored with most current stuff and
  • now produces an error. That's what another guy's post a week or so ago
  • indicated. I think he had to stop setting local symbol. I'm too lazy to
  • look back. 😉
  • Anyway what was really curious was to see the local symbol coming back in
  • your updatePortfilio message, per your report. I probably never display the
  • field for FX so maybe I should take a look, or maybe it is too late now
  • since I am at 9.64 api with the latest TWS which I think is 905.
  • -Kurt

[Q] IDEALPRO's hours?

[A] by climbon6

IDEALPRO's hours are 17:15 – 17:00 (ET) Sunday-Friday.

The market was closed when you got the Bid price = -1.0

http://interactivebrokers.com/en/trading/exchanges.php?exch=ibfxpro&showcategories=FX&ib_entity=llc

FX-related note:

Yes I request BID, ASK, BID_ASK, and MIDPOINT work. IB does not offer TRADES for FX.

[Q] Minimum size requirements for IDEALPRO

[A] by Orionn

there are minimum size requirements for IDEALPRO (trades) whereas

for IDEAL (currency conversions) there are not. If the minimum size is

not met for an IDEALPRO order, it is moved to the IDEAL order book. This

is for real trading accounts, paper trading accounts and demo accounts.

[Q] Minimum Price Increment

[A] by Josh see full thread

Added on 27-Feb-2021

There is a setting in TWS at Global Config -> Display -> Ticker Row -> "Allow Forex Trading in 1/10 pips" which is not enabled by default.

http://interactivebrokers.github.io/tws-api/minimum_increment.html

The default is 1/2 pips. Settings are saved separately between live and paper accounts.

Update: There is also a bug in TWS relating to this setting: at least, there was in TWS 972 at the end of May – it might have been fixed by now, but I thought I’d mention it in case it’s still there.

 

To see the bug, set the "Allow Forex Trading in 1/10 pips" checkbox, and save the settings. Then open the Global Configuration dialog again and check the setting: it will still be checked. Now go to the API Settings page in the Global Configuration dialog, and close the dialog (you don’t have to make any changes in the API Settings page, just opening it is enough). Now go to the “Allow Forex trading in 1/10 pips” checkbox again, and you’ll see that it has been cleared.

 

In other words, opening the API Settings page will reset this checkbox to unchecked.

 

Richard


Glossary

NMS is apparently National Market System

SCM = Small-Cap Market


Other

[Q] Using Managed Accounts: How to set deposit size on subaccount / disable loan?

Hi

My main IB account uses some margin loans.

Is it possible to open a connected account, to be used with TWS API, which can't take any margin loan (but keep the main account enabled to loans)?

For example, I have –

On my main account, stocks valued 100,000$ and cash value of (-10,000)$ – I deposited 90,000$ and took 10,000$ loan.

On my connected account I would like to deposit 1,000$, and disable loans for this account.

So, for example, if I try to buy a stock on the connected account, and it has only 500$ cash, and the stock costs 600$ – the order would fail.

So there can't be any situation in which I loss more then 1,000$ using TWS API.

And on my main account, I will still be able to take loans as always.

Is it possible? How can I set it up?

[A] by Dan View/Reply Online (#48661) 

Added on 21-Dec-2020

Yes, you can add multiple accounts and configure each account separately.

To add an account: https://www.ibkr.com/faq?id=23299014

To configure the account as a Cash account: https://www.interactivebrokers.com/en/software/am3/am/settings/accounttype.htm

To configure trading permissions: https://www.ibkr.com/faq?id=27271500

And: https://guides.interactivebrokers.com/am3/am/settings/tradingpermissions.htm

Update: View/Reply Online (#48673)

Thanks!

What happens if I try to place an order on such account, and I don't have enough cash for it? The order would just cancel?

You will get an error message stating that your order is refused, due to insufficient cash. The word "cancel" is only used after an order is accepted and processed and then cancelled. Your order would be refused and therefore not processed.

[Q] Best take profit order type

[A] by alpha pmularien@gmail.com View/Reply Online (#44725)

Added on 18-Jun-2020

My strategy is:

  • Set a limit-if-touched order (with appropriate limit values based on typical spread for the security) to catch any unexpected spike in price (before my algo may catch it)
  • When the price approaches my target, replace the LIT order with a trailling stop limit (TSL) (with appropriate trail/limit values based on the typical spread for the security)

TSL orders work pretty well, but like any stop limit order, you are vulnerable to the price falling through the stop, therefore the limit is really important to ensure you have a better likelihood to capture the trade at/near the price. Make sure you set the start price on the correct side of the spread 🙂 — below the spread for sell, above the spread for buy.

[Q] contract->exchange = "SMART" works, but contract->exchange = "NASDAQ" does not!

[A] by Angel (IB support)

exchange = NASDAQ is not valid, to use the NASDAQ Exchange you must specify the ECN which is ISLAND.

Regards,

-Angel 

IB API Support 

[Q] IB technical support contact info?

[A] by 'ce' zadig_1 at excite.com

Added on 18-Feb-2016

information you are looking for is in Account Managent, support , contact :

 

United States        1 (877) 442-2757 Toll free

1 (312) 542-6901 Direct dial        08:00 – 20:00 (Mon – Fri) (America/New York)

13:00 – 19:00 (Sun) (America/New York)

 

   1/5 Technical Assistance

      1/5/1 Software Installation & Connectivity

      1/5/2 Software Operation

      1/5/3 Software Functionality

      1/5/4 API Support

Also Broom IAmBroom at gmail.com:

You can also reach them via Chat, but telephone is really the best way to go. IB discontinued email support completely a long time ago.

[Q] Fully clarifying exchange vs primaryExchange

[A1] by 'Richard L King' rlking@aultan.com

Added on 03-June-2016

Liu

When you use a Contract object in the TWS API, you are really defining a

query on IB's contract database. This contract database contains a huge

number of different securities, so if you're only interested in one of those

(for example when you place an order) you need to make sure that the query

expressed by your Contract object identifies a single security.

Now, if you're placing a directed order for a stock, ie to a specific

exchange, then this is usually pretty straightforward: the symbol, the

sectype, the multiplier and the exchange will be enough (for a stock) to

ensure uniqueness (but I don't doubt someone will point out a case where

that fails and you need to supply something else like tradingClass!).

But if you want to send a SMART order, there is a potential problem. You set

exchange="SMART", but IB has more than one SMART routing infrastructure.

Many stocks are traded in both the USA and Europe, and IB has different

SMART routing infrastructures for those two continents (and possibly for

Asia as well).

So you have to supply information that tells IB which SMART router to use:

you can generally do this just by specifying the currency, because stocks

traded in Europe are normally traded in Euros, and those in the US are

traded in USD. But if for some reason you can't or don't want to specify the

currency, then you can specify primaryExchange (instead of or in addition to

the currrency): so primaryExchange="NASDAQ" tells IB to use the USA SMART

routing infrastructure for your order; similarly for your market data

request, it tells IB to send you the best bids and offers across all the

exchanges in the USA SMART routing infrastructure.

So to summarise, primaryExchange is only ever needed when exchange=SMART,

and even then may not be needed if you've supplied enough information in

your contract object to uniquely identify which SMART routing infrastructure

you need.

I probably ought to add that this was certainly the original purpose of

primaryExchange, but possibly it may now also have other uses, so I'd be

interested to learn if anyone has further information on this.

By the way, it's very worthwhile to write a little utility that enables you

to enter the details for a contract query, do a reqContrqactDetails, and

display all the returned contract details in an easily-readable form. This

makes it very easy to find out what's going wrong when you get those nasty

error 200s and the like.

RIchard

[A2] Also read this thread

I will copy-n-paste the whole thread here due to its high importance:

——————————————————————————

This has come up before as a topic, but the response was not quite explicit.

I have made my own assumptions about exchange and primaryExchange that have

worked for years. But I think I have gotten away with sloppy thinking and

would like to make this understanding fully explicit.

We often discuss how primaryExchange is used when necessary if the contract

is otherwise ambiguous, e.g. usually unnecessary when currency is specified.

This encourages the thinking that primaryExchange is oriented toward

specifying the contract.

In the case of stocks/options anyway the exchange field is something I think

of as oriented toward the notion of an order *or* a request for market or

historical data being "directed". This encourages the thinking that the

exchange field is used for "directing". To not "direct" use "SMART" (when

applicable).

In summary so far that suggests a distinction between exchange and

primaryExchange being a distinction of "directing" (order or data source)

versus "specifying" (contract).

In the case of futures (which I don't use myself so far) it seems that

specifying the exchange is usually or always required, e.g. GLOBEX. This

encourages the thinking that for futures, exchange is needed to specify the

contract, if I understand futures enough to make a sensible statement.

That creates two different points of view.

One answer to this question in the dim past pointed to "SMART" but only as

an example, not as a rule. If you specify exchange "SMART" you may need to

specify primaryExchange in case of ambiguity.

It may simply be then that you may specify an exchange for "directing" if

desired, and *only* if that leaves an ambiguity then you must specify

primaryExchange to resolve that ambiguity. That point of view resolves the

conflict above and suggests that exchange and primaryExchage may work

together to "specify" (as needed) and that exchange may be used for

"directing", with a side-note that if exchange is "SMART" (or some other

"aggregate"?) some expressiveness was lost in the use of the exchange field

to "specify" and that therefore the primaryExchagne field is simply a hack

to permit an exchange to be specified when there was a desire *not* to

specify an exchange for "directing" purposes.

Does that last model turn out to be correct? If not has someone come up

with something like a formal explanation of these two fields that can

relieve me of my sense of having been sloppy about it (and as such unable to

feel comfortable about certain design decisions in my UI)?

Though I've never traded futures I've tested my symbol-entry UI with them

and I use the exchange field, not primaryExchange. I wonder if it would be

possible to use primaryExchange instead, or does that depend on some

"aggregate" being specified for exchange, and does no such aggregate exist

for US futures?

Thanks.

-Kurt

——————————————————————————

Kurt,

I have the same understanding: Exchange stands for data source, while primaryExchange is a hack to resolve the ambiguity in the triplet (symbol, exchange, currency) which only a few stocks suffer from. PrimaryExchange only helps "specifying" a contract and not "directing" to/from an exchange.

That hack may prove obsolete with API 9.69 since conId will also resolve ambiguities.

For futures, using exchange="SMART" and primaryExchange="GLOBEX" makes little sense, as there's no aggregate source (as far as I know) and the output is the same as simply exchange="GLOBEX"; that syntax only works since API 9.69 while older APIs give "no security definition".

souqMate

——————————————————————————

If you are saying "SMART" is now accepted (in the API only?) for futures, I think it makes good sense and improves consistency.  If "SMART" could always mean undirected and exchange only used to "direct" that is good, could reduce headaches for beginners, and permit developers of general-purpose UI to use trivial instead of hard-to-rationalize (per my post) logic to map input fields onto internal fields.

It is called "orthogonality" I think and it is a good property of relationships among communicating entities.  Let "SMART" be trivial when there is no aggregate, fine.  It is still consistent, mathematically useful, so to speak, yields good programming.

So if this true I will code that way and use an appropriate input syntax that will work for everything.

But for completeness, to be sure I understand currently-accepted behavior, I should confirm that using NASDAQ as exchange for INTC (in directed mode) permits me to omit the otherwise newly-required primaryExchange.  You can answer if you like.  🙂   And otherwise I will try it soon.

-Kurt

——————————————————————————

If you specify exchange to be anything else than SMART, you cannot on top specify pirmaryExchange (else get error 200 "Invalid destination exchange"); true for API 9.69 or older.

You can use exchange="ISLAND" (stands for Nasdaq) and omit primaryExchange="NASDAQ", but for historical data you'll miss out liquidity from other exchange.

What I do for stocks is to always set exchange="SMART" and primaryExchange="…" (sourced from a config file); that helps me avoid your surprise with the stock INTC.  I don't even have to amend my config file when a stocks moves from Nyse to Nasdaq (eg FCS a month ago), as primaryExchange can have any value (NYSE,NASDAQ,AMEX,…) as long as the triplet (symbol,exchange,currency) has pinned down a single contract; true for API 9.69 or older.   Since API 9.69, I could source conId from my config file instead of primaryExchange.

souqMate.

——————————————————————————

sorry to disturb your discussion.

where can I find more details about exchange and primary exchange explanation? or maybe meaning of these "SMART" and possible combination?

Thanks.

——————————————————————————

Answer from api@interactivebrokers:

4. What's the difference between the Exchange and Primary Exchange fields? in what cases are they used?

The primary exchange is the main exchange on which a given stock is listed. The stock may trade on other exchanges as well, but it will always have one and only one primary exchange. You can see it as part of the contract's definition. You can ignore it and use Exchange alone. 

5. What type of securities do SMART routing apply? Does it only work on the Exchange Field?  

-Yes, Smart routing refers to the exchange the order will be routed to. You can find more detils about it at http://individuals. interactivebrokers.com/en/ index.php?f=1685 

——————————————————————————

I suppose we need to decide for our apps whether we want to require a specification step in the UI before trading a symbol, or if we want to handle the exceptions (in which the exchanges used for "SMART" have multiple interpretations for the same symbol) by bringing up additional UI only in those cases. Or make the symbol field long enough to enter something like "INTC@NASDAQ" . (One might need something like that also for stocks traded on foreign exchanges that are not in the SMART-supporting group of exchanges).

The current "old" API (without using conId) allows a simpler UI and implementation for many (who are trading mostly only stocks and stock options), which might be useful (in a somewhat limited way) since the exceptions have probably been rare for self-use (although INTC perhaps changes that, and other new contracts).

——————————————————————————

I already had the syntax symbol@exchange with exchange defaulted to "SMART", so I extended it to be symbol@exchange@primaryExchange, which can simply be symbol@@primaryExchange as needed for example with INTC@@NASDAQ.

This approach brings the full idiosyncrasies of the API to the surface in the UI syntax, letting the "user" deal with it.

-Kurt

——————————————————————————

Right, given you apparently support directed trading (for which I haven't yet discovered the advantage).

I think the idiosyncrasy would be that the "exchange" field is in the Contract, vs. in the Order. I take it to mean "defaultOrderExchange". Except it also narrows down the possible ambiguities in terms of the search criteria that IB will use to identify the contract. I wonder if using the conId leads to faster order execution. Might test that soon.

——————————————————————————

With the "latest" production versions I am using, specifying the conId (270639 for INTC) and "SMART" is enough. I suppose this is the most efficient way. reqHistoricalData still complains if I don't specify an exchange, but "SMART" is enough, it doesn't have to be a specific exchange. Even currency and "STK" are apparently not necessary.

——————————————————————————

> I take it to mean "defaultOrderExchange". Except it also narrows down the possible ambiguities

Not only that.  It determines the source for market data and historical data.  So it really has to be in the Contract.

You will see this matters on those occasions when an exchange "breaks", showing bad quotes which then pollute "SMART" market data.  When that has occurred I have intervened manually and picked a particular exchange to receive quotes from, while still submitting orders as SMART.

-Kurt

——————————————————————————

> Not only that.  It determines the source for market data and historical data.  So it really has to be in the Contract.

So it would be more precise to say that the "exchange" field could be thought of as "exchangeToTradeOn". As exchange-specific market data and historical data can be seen as part of trading.

And this also clarifies why it is still necessary to specify an "exchange", even if just generally "SMART", when using the conId. None of the other fields (not even "currency") are necessary when the conId is specified. (In the "latest" versions, testing with reqHistoricalData).

> You will see this matters on those occasions when an exchange "breaks", showing bad quotes which then pollute "SMART" market data.  When that has occurred I have intervened manually and picked a particular exchange to receive quotes from, while still submitting orders as SMART.

They just "break" once in a while? Oh my. 😉
——————————————————————————

P.S.: I doesn't seem that "primaryExchange" is just a "hack" to resolve ambiguities, in the sense that it seems a valid way to distinguish multiple results returned by reqContractDetails when those have the same currency, for example. It's more "currency" that would be a hack in that regard. But it's probably easier to know the currency you want, than to know the primaryExchange you want.
——————————————————————————

It is what I might call a "hack" to the degree that the interrelated functionality is not orthogonal or consistent in behavior.

I was hopeful for orthogonality until souqMate cleared it up (3rd and 4th posts in this thread), saying:

    If you specify exchange to be anything else than SMART, you cannot on top specify pirmaryExchange (else get error 200 "Invalid destination exchange"); true for API 9.69 or older.

which may only be a passing problem.  But passing problems will prevail in the end.

-Kurt

——————————————————————————

Yes, to me that looks like a bug in the implementation on the server side. It should always be allowed to specify the correct primaryExchange to ensure that the intended equity is used. Especially the combination (exchange=ISLAND,primaryExchange=NASDAQ) should be valid, but in fact I also get errorCode 200, like souqMate. I'd say we should bring this to IB's attention. I guess as a workaround, one can omit primaryExchange when using anything else than SMART for the exchange. Unless there is a surprising explanation.

——————————————————————————

[A3] Also another more recent discussion on "exchange" vs "primaryExchange" from January 2020:

https://groups.io/g/twsapi/topic/69604228#43521

I'll copy-paste here since it's valuable

Difference of "primary exchange" and "exchange" ?

 Richard L King

Jan 10   #43511

Primary exchange is used to disambiguate two (or more) similar contracts.

To take the MSFT example, if you request contract details for MSFT without specifying the exchange (ie just use symbol=”MSFT” and sectype=”STK”), you get 25 contracts returned. Two of them have exchange=”SMART”, so if you want to SMART-route an order you have to supply something additional, and primary exchange is one way of doing this: if you like, it’s a hint to IB to tell them which SMART routing infrastructure to us: the US one (primary exchange=”NASDAQ”, or the European one (primary exchange=”EBS”).

 

Primary exchange is primarily relevant when using SMART as exchange in your order placement, market data requests, etc. Since SMART can only be used with stocks and options, you probably never need to supply primary exchange for other securities. And for stocks and options where you are direct routing to a particular exchange, primary exchange is probably completely irrelevant. (I say ‘probably’ because nothing ever seems to be certain in the realm of contracts!…)

 

There are often other ways to do this disambiguation (for example different currency codes).

 

It’s worth while exploring contracts by trying various specifications and seeing what you get back. I wrote a program called the Contract Inspector that does all the hard work and is very useful for trying to understand this stuff. It’s free and is available (Windows only I’m afraid) on GutHub at https://github.com/tradewright/ibapi-tools/tree/master/ContractInspector. You can install it directly from http://www.tradewright.com/contractinspector/.

 

By the way, playing with the Contract Inspector soon reveals many oddities about IB’s contract request implementation, as you’ve been discovering for yourself! Often the answer to ‘why does it do this in situation A, and something different (or nothing at all) in situation B’ is: who knows?!

 

[ Someone else on this group (btw I think?) produced a Java version of the Contract Inspector by translating part of my C# code and adding a JavaFX UI, but I don’t recall where it is, and I have no idea what its current status is. Perhaps that person might want to clarify. ]

 

Richard

 Richard L King

Jan 10   #43512  

It’s probably worth adding that having been developing my trading platform since 2003, it was only in the last year that I started using primary exchange at all, and that was when I stated doing some serious work using options. I don’t recall the details, but I suspect I came across some options contracts that could only be disambiguated using primary exchange.

 

Before that I had never found a valid reason to use primary exchange – there always seemed to be better solutions available, but it probably meant that there were a few contracts that couldn’t have been accessed via my platform.

 

R.

 alf

Jan 10   #43513  

Thanks for the clarification.

One question: MSFT is officialy listed on NASDAQ and not on NYSE, so I expect that using this query:

Contract c = new Contract();

c.symbol("MSFT");

c.secType(SecType.STK);

c.primaryExch("NASDAQ");

c.currency("USD");

or this query:

Contract c = new Contract();

c.symbol("MSFT");

c.secType(SecType.STK);

c.exchange("NASDAQ");

c.currency("USD");

returns the appropriate contract. But IB does not find any contract with one of those queries. Only if I replace "NASDAQ" to "NYSE", it finds the right one and then returns MSFT with "m_primaryExch" set to "NASDAQ". So why does IB did not find MSFT with "NASDAQ"?   

 Rwk

Jan 10   #43514  

On Fri, Jan 10, 2020 at 10:18 AM, alf wrote:

One question: MSFT is officially listed on NASDAQ and not on NYSE

I trade USA stocks using an external database that provides a symbol universe, symbol changes, and price adjustments for splits and share dividends.  I see data as a utility, like the electric power, and I don't want to have to think about it.  My database is Telechart TC2007, but there are others that work okay.

I use "SMART" for exchange and "ARCA" for primary exchange.  I do preliminary scans in Telechart, then update that set with realtime data from IB.  I have never failed to get IB data for the chosen symbols this way.  It's not perfect, but it's cheap, reliable, and easy to program.  I have done extensive testing to study quality of order fills and deviation from theoretical performance, and I am comfortable with the tradeoff between accuracy and cost & convenience & reliability.

Everybody has their own approach, so I am not saying mine is best.  I have found that trading profitably is a lot harder than it looks, and building infrastructure is a big distraction from what is most important.  I try to keep it simple and control costs.

Good luck … the answers are out there!

[rwk]

 Richard L King

Jan 10   #43515  

First, as far as I’m aware you can never use “NASDAQ” for the exchange field. I’m very unclear as to the reasons for this, but it’s really irrelevant.

  • EITHER you use “SMART” to have your order smart-routed (in which case you can supply a primary exchange field that informs IB to se the US smart routing infrastructure, and as rwk has just pointed out, using “ARCA” as primary exchange always works, as does “NASDAQ”, “AMEX”, “BATS”, “CBOE” etc);
  • OR you specify a specific exchange such as ISLAND, and you don’t supply a primary exchange.

 

So in your first case, you haven’t specified an exchange at all, so I guess IB doesn’t know what you mean. In your second case, you’ve supplied an incorrect exchange – note however that if you supply a nonsense exchange like “BOZO”, you get the error message ‘Invalid destination exchange specified’, so it clearly recognises “NASDAQ” as an exchange name, just not one it considers relevant.

 

So like I said, who knows?

 

 

toggle quoted message

Show quoted text

 alf

Jan 10   #43516  

Thanks.

This works:

Contract c = new Contract();

c.symbol("MSFT");

c.secType(SecType.STK);

c.exchange("SMART");

c.primaryExch("NASDAQ");

c.currency("USD");

It seems that NASDAQ is too unspecific to set as "exchange", so it must be more specific like "ISLAND". However, to avoid the use of "primaryExchange", it seems the best to use the following, which returns the same result as above:

Contract c = new Contract();

c.symbol("MSFT");

c.secType(SecType.STK);

c.exchange("ISLAND");

c.currency("USD");

 Nick

Jan 10   #43517

I'm not sure what your aversion to primary exchange is.

You should be deciding what kind of routing is appropriate for your strategy and use that in the api.

Smart routing is not the same as direct routing. It makes no sense to choose a routing based on what fields you have to populate in the api.

 Richard L King

Jan 10   #43518  

No, it’s not the same.

Your first contract is smart- routed, which means an order placed using it will be executed on whichever of the North American  exchanges has the current best bid/offer.

The second will only be executed on ISLAND, so you may not get the best possible fill.

It sounds like you need to gain a better understanding of IB’s SMART routing abilities.

Richard

 

 alf

Jan 10   #43519

Okay, got it. Thanks!

I actually did not dig into the ordering procedure. At first, I only wanted to get the common details of a contract by reqContractDetails()-method with by providing "symbol", "security type", "currency" and "exchange". However, as you see: the exchange must be treated special in IB because it can be SMART-routed.

Thanks again for the clarification!

 alf

Jan 10   #43520  

I found this little sentence in the official API documentation (https://interactivebrokers.github.io/tws-api/basic_contracts.html)

//In the API side, NASDAQ is always defined as ISLAND

Does this mean, that in the API side, we should always put "ISLAND" in exchange or

primaryExch to express "NASDAQ"? So if I want to say NASDAQ, I must use "ISLAND".

Yair Altman

Jan 11   #43521  

Almost correct – If you use the API, you should indeed not use Exchange='NASDAQ', but rather 'ISLAND' (or 'SMART'). However, this apparently applies only for stocks and other tradeable contracts. I believe that if you use SecType='IND' then you can/should indeed use Exchange='NASDAQ'. Confusing indeed, certainly undocumented, and subject to change by IB's API server at any time without prior notice. Who said API programming isn't fun? 🙂

 Reply

 Like

 More

 misantroop

Jan 16   #43579  

What's even more confusing is that some requests like reqPositions return the primary exchange as 'exchange' and not 'primary_exch' like everywhere else which is inconsistent as the contract objects cannot be moved around freely as they should be. Querying IB resulted in the usual "it's a feature, not a bug" reply. Easily fixed locally but messy and confusing nonetheless.

 Reply

 Like

 More

 1 person liked this

[Q] I need some actively trading stock for my midnight debugging

[A] by Dmitry

check out open markets: http://www.worldtimezone.com/markets24.php

or: http://www.isthemarketopen.com/

say you choose Hong Kong Exchange, now use IB contract search engine:

Contract and Symbol Database – Interactive Brokers

or google most active symbols from Hong Kong exchange(s).

[Q] Where do I get the latest standalone TWS?

[A]

for mac: https://www.interactivebrokers.com/en/index.php?f=674&os=mac

time-tested version”: TWSX_install.pkg

TWS 9394 • May 5 2014

or

TWS Latest is the most current version”: TWSX_install_latest.pkg

TWS 9443 • Apr 30 2014

[Q] How do I know if the market is open via API?

[A] by skunktrader2001 (from this thread)

You could request the ContractDetails and parse the field called m_tradingHours to get the times that the market is open

example: for SPX I got: 20101202:0830-1500

[A2] by skunktrader2001

Another field in ContractDetails is m_timeZoneId. You can easily use that to initialize an instance of java.util.Calendar to convert exchange local time to "desk time right now". The advantage is you can use IB to keep track of holidays / early closes.

That being said, my own program has a local database where I manually keep track of holidays / abnormal session open / close times.

[Q] How to get account balance, transaction fees, cash transactions, Net Asset Value, etc.? (Flex?)

[A] by andy_flury

You can use the Flex WebService to retrieve account balance, transaction fees, cash transactions, Net Asset Value, etc.

Just go to your Account Management / Delivery Settings / Flex Webservice. There you generate a token. With that token you can retrieve an nice XML containing all the information you need.

I retrieve this XML once a night and update my database.

cheers

andy

———————- also ————————

I just uploaded my Java-Class to the files section / Java_Code / CashTransactions.java

Hope it helps.

Andy

———- also Dmitry’s notes:

as of Jan-2014, IB changed GUI, so to create new Flex Reports we go to:

Reports -> Activity -> Flex Queries

and for token management:

Reports -> Settings -> Flex Web Service

by default they’d generate token with less than 1 week expiration, but you can re-generate one wit up to 1yr.

[Q] How to automate Flex Reports?

[A] by Eric J. Holtman (from this thread)

I was able to get the web flex delivery working. I've pasted a

small bash script at the bottom which seems to work. I tried to

do this last night, but there was a typo I couldn't see.

If you're running cygwin with wget installed (or running Linux),

this seems easier than trying to get something out of email,

and you can set it up to run without human intervention.

This skeleton script does no error checking at all.

# your secret token from IB's flex webserice goes here

TOKEN=2584375894759837598437

# pass query id for whatever query you want as the only argument

QUERY_ID=$1

# I'm using –no-check-certificates only because the version of cygwin

I'm using

# has an old cert file, there is nothing werong with IB's actual certs.

# first issue the request to generate the report and return a query code

# pipe through a brain damaged sed script to extract the magic code

QUERY_CODE=$(

wget -q –no-check-certificate -O –

"https://www.interactivebrokers.com/Universal/servlet/FlexStatementService.SendRequest?t=$TOKEN&q=$QUERY_ID"

|

sed -n -e 's/<code>\(.*\)<.*/\1/p'

)

# now issue the request for the actual report, dump it to stdout

wget -q –no-check-certificate -O –

"https://www.interactivebrokers.com/Universal/servlet/FlexStatementService.GetStatement?q=$QUERY_CODE&t=$TOKEN&v=2"

Reply

[Q] Which Java is better to use for TWS on Ubuntu?

[A] by Mike Smith

I get the same exception in Ubuntu and I have no problems with the TWS window disappearing so I doubt that is related. If you are using the Java that comes with Ubuntu 10.10 (OpenJDK) then you might try using the Sun/Oracle JVM instead. OpenJDK has been known to have problems.

http://aroundtheweb.info/2010/10/install-sun-java-ubuntu-10-10-maverick-meerkat-official-partner-repository/

Then you can switch between JVM's using:

sudo update-alternatives –config java

The best thing about using the Sun/Oracle JVM is that it is updated more often by Ubuntu's Update Manager. I have not had an update to OpenJDK ever.

Mike

[Q] Is there any way to see S&P 500 index on TWS?

[A] by Abhijit Dey

why not? I am getting quotes for SPX. It's an additional subscription

for me (professional sub.). All of $3 extra per month for CBOE

Correction. It's $2 per month. It's called "CBOE MARKET DATA EXPRESS

INDICES PROFES"

[Q] Release notes – should I read it?

[A] by Kurt

I think this may be documented in the release notes. There

definitely was a time (somewhat less so now) when the release notes had to

be considered part of the API documentation since much information was there

that did not appear anywhere else. Unfortunately some of the info in the

release notes was incorrect, and old notes are never corrected. Also new

approaches sometimes supersede older ones. What I usually suggest is that

the release notes should be read in their entirety from oldest to newest, a

couple times. IB makes this hard because you can't go *back* to the main

release notes page directly. I found it worthwhile to print the release

notes. I'd suggest going back 3 or 4 years, if you want to learn the API.

[Q] French transaction tax, how to anticipate?

[A] by bgpl.mirror@gmail.com

http://ibkb.interactivebrokers.com/article/1963

or google:

site:interactivebrokers.com "Shares Subject To The French Financial Transaction Tax"

[Q] Getting exchange location for order?

[A]

  • It looks like I get an openOrder and orderStatus callback whenever
  • an order is rerouted in TWS…
  • that is, if I specify "SMART", but watch in TWS, I get those
  • callbacks when the order gets recalled and redirected to a
  • new specific exchange. (like, "off DIRECT, now sending to NYSE").
  • I've looked through the callbacks, and I can't see anywhere
  • I can capture this info… does anyone know if it's available.
  • Obviously, TWS is getting it. But that of course doesn't
  • mean the API is.
  • Reply
  • Kurt Bigler
  • Jun 1, 2010
  • View Source
  • The openOrder callback has a Contract argument. Did you look there?
  • -Kurt
  • Show message history
  • Reply
  • Eric J. Holtman
  • Jun 1, 2010
  • View Source
  • On 6/1/2010 1:48 PM, Kurt Bigler wrote:
  • >
  • > The openOrder callback has a Contract argument. Did you look there?
  • >
  • That only ever has SMART in the exchange field (which I filled in),
  • and nothing in the primaryExchange field.
  • The execution report does tell you where, but (at least I think),
  • I can see TWS updating even if I don't get an execution.
  • Not a huge deal, just was wondering if I could catch it because
  • then I could later track it against regional exchanges BBO.
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • … Probably not what you want to deal with, but I m wondering whether the audit trail files contain transactions for routing events. It seems like they
  • Jun 1, 2010
  • Eric J. Holtman
  • Jun 1, 2010
  • View Source
  • On 6/1/2010 3:50 PM, Kurt Bigler wrote:
  • >
  • >
  • > Probably not what you want to deal with, but I'm wondering whether the
  • > audit
  • > trail files contain transactions for routing events. It seems like they
  • > contain such an abundance of trivial stuff, so maybe this would be
  • > included,
  • > but I did not look for it.
  • >
  • I was looking through the audit trail, and it does not seem
  • like they are there…. I only see the barest of FIX messages,
  • not the plethora of OrderStatus ones that I really want to see.
  • I'd have to pay closer attention and really track it, it's
  • entirely possible that what I'm seeing in TWS isn't what
  • I think it is, it might just be the exchange of the last exec.
  • I only happened to notice today because the paper
  • account seemed to be broken, not giving fills, for a while,
  • and I thought that during that time, I saw the TWS
  • changing the "destination" field without getting a
  • partial fill.
  • But maybe I was just high.
  • [Non-text portions of this message have been removed]
  • Reply
  • Kurt Bigler
  • Message 6 of 6 , Jun 1, 2010
  • View Source
  • On 6/1/10 2:56 PM, "Eric J. Holtman" <eric@…> wrote:
  • > On 6/1/2010 3:50 PM, Kurt Bigler wrote:
  • >> Probably not what you want to deal with, but I'm wondering whether the
  • >> audit
  • >> trail files contain transactions for routing events. It seems like they
  • >> contain such an abundance of trivial stuff, so maybe this would be
  • >> included,
  • >> but I did not look for it.
  • >
  • > I was looking through the audit trail, and it does not seem
  • > like they are there…. I only see the barest of FIX messages,
  • > not the plethora of OrderStatus ones that I really want to see.
  • >
  • > I'd have to pay closer attention and really track it, it's
  • > entirely possible that what I'm seeing in TWS isn't what
  • > I think it is, it might just be the exchange of the last exec.
  • No I've definitely seen the routed exchange appear, lots of times. 99 times
  • out of 100 for me I get filled in one execution, so I wouldn't be seeing the
  • last exec.
  • I just placed a real order (in TWS) at fussy price to test it. It showed
  • destination as SMART prior to transmit. After transmit it quickly
  • (immediately as far as I could tell) changed to ISLAND. I'm not 100% sure
  • I've seen the destination change *later*, but I think so, maybe with option
  • combos. Regardless this is a piece of info that is showing in TWS that from
  • what you say is not available via the order status Contract.
  • But I see the audit trail does show the destination. Maybe the original
  • destination is different from a re-assigned destination. Anyway the trail
  • looks like this — see ExDestination under Acknowledged:
  • PlaceOrder 20100601-22:34:02 1223997357.0 – Buy 5 AAPL
  • Limit 257.00 DAY Account= U383859; 6240 = 1; 6408 = 18409420238;
  • OptionAcct= c; 6433 = 1; 6419 = IB; 6532 = 20100601-22:34:02; SecurityType=
  • CS; ContractMultiplier= 1.00; ExDestination= BEST; 6210 = BEST; ContractID=
  • 265598; Currency= USD; 6211 = ; 6238 = ;
  • Acknowledged 20100601-22:34:02 1223997357.0
  • 0003bfe1.0000eee7.4c049011.0001 Buy 5 AAPL Limit 257
  • DAY ExecID= 61159.1275431642.1; ExecType= New; ExecTransType= New;
  • OrdStatus= New; ExDestination= ISLAND; SecurityExchange= ISLAND; CumQty= 0;
  • LeavesQty= 5; AvgPx= 0; SecurityType= CS; Account= U383859; 6532 =
  • 20100601-22:34:02; TransactTime= 20100601-22:34:02; 6571 =
  • 20100601-22:34:02; Currency= USD; Exchange= BEST; OptionAcct= c;
  • CustomerOrFirm= Customer; IBLocalSymbol= AAPL; 6674 = 0;
  • I would think the audit trail would need to show a re-assigned destination.
  • It is sort of your contract in case of execution failure at the exchange.
  • -Kurt

[Q] What is the "numIds" for in reqIds function?

[A] by Werner Stumpe wstumpe@gmx.de

If you know that you need, for example 5, reqIds, you set the parameter

numIds to 5. What you get is the highest of 5 consecutive reqIds, and

they are reserved for you. So, if you get 44 as the reqId, you have 44,

43, 42, 41 and 40 reserved for you.

wstumpe

[Q] Does anybody know of a free technical analysis libraries?

[A] by Abhijit Dey

        TA: http://ta-lib.org/index.html

     quant: http://quantlib.org/index.shtml

[A2] by Gord

Google: CCAPI. 2001ff, activestocks.de / Ulrich Staudinger

Look for the file FinancialLibrary.java its readable.

[A2] by MarketMole

http://www.eclipsetrader.org/

[Q] Is there a way to pull these daily libor rates directly from TWS?

[A] by Kurt

Various benchmark rates are available from this page:

http://www.interactivebrokers.com/en/accounts/fees/interest.php

I don't know how often the rates are updated or whether you care about

having the new rate the very day it changes. But code can certainly be

written to extract these rates from the web page above, or conceivably from

numerous worldside sources of similar rates published on the web, in case

you find that the IB page is not current enough.

So you load data from the appropriate url, and parse it to get what you

need, using whatever TCP and parsing libraries you have available to you.

-Kurt

[Q] Closing Prices in TWS Are Wrong?

(found here)

[A1] by Kurt Bigler

I don't know about futures, but the Close column of TWS presents some

confusing information after hours for stocks and stock options–but nothing

quite like what you described so soon after closing. The close will always

be yesterday's close until some point after extended hours trading closes,

at which point "today becomes yesterday" and you see the new close, which is

of course the close of regular trading, not after-hours trading. During

some period of times I recall seeing the Close price jump around a bit,

related to these daily transitions as I recall.

When I complained to IB about it they stated that they are just passing on

the raw information from the exchanges, and you get what you get. So for

stocks, I've just gotten used to it.

Options are worse than stocks, because what eventually appears is another

thing IB says is the "settlement price" from the exchange, but it is hard

for me to imagine how a settlement price can be so far away from the close

as it often is, with no (to me) imaginable relation to what happenned that

day, not even as a volume-weighted average price. Online info from CBOE

etc. that I've been able to find about "settlement price" is entirely

unhelpful. But I'm way off your topic now.

You might check whether historical 1-day bars are any more helpful for

getting a meaningful close price for a particular day. Be sure to specify

RTH=1 for what I think you want.

-Kurt

[A2] by Eric J. Holtman

I used to trade options on futures pretty heavily. The "settle"

they would send out would, as you say, often have no relation to

any action during the day.

In fact, many times, we'd have orders on the floor for a combo,

would (of course) get no nibbles, and then at the close of the

day, they'd settle both legs through our prices.

Now, in their "defense", they weren't picking numbers out of a

hat….. if you graphed the settle prices for the S&P 500

futures options, you'd get a volatility smile that was so

smooth you'd swear the data wasn't real.

So, clearly, their methodology was to decide how the vols

moved (direction and shape), and then peg settlement to that.

[Q] How to place MOO orders?

[A] first the question in its wide form:

I was hoping someone could help me with a couple novice questions on getting the correct order settings for submitting Market-On-Open orders.  I have followed the instructions in the user guide and tried to get answers a few times via IB chat, but thus far have not been able to get a set of instructions that works for all cases.  Essentially, all that I want to do is participate in the opening auction/cross for symbols that have either NYSE, NASDAQ, AMEX, ARCA in their TWS description.  For example, I was told that for ARCA symbols I just needed to direct route to ARCA and use MKT and OPG — however, after selecting ARCA in TWS, the OPG option is not available in the TIF drop down.  For NASDAQ, I was told to direct route to ISLAND and use MKT and OPG, but orders submitted that way executed immediately (prior to 9am EST) rather than waiting for the ~9:30 open.  

  • If anyone would be willing to share a working set of order settings (routing, TIF, anything else relevant) for each of the 4 listings I would greatly appreciate it.  
  • Also, if I might ask one additional question: assuming that these orders will be to initiate new positions, will the handling of odd lots/mixed lots be different than what IB has on their site for typical orders (i.e. direct routed odd lots will be rejected and the partial lot of a mixed lot will be canceled after the round lot is executed)?  Thus far I have only tested round lots.  
  • Thanks in advance for any input.  
  • [Non-text portions of this message have been removed]
  • Reply
  • Rob Terpilowski
  • May 28, 2010
  • View Source
  • For MOO orders I exclusively use MKT with a TIF of OPG, and leave the routing as SMART, IB will then route the MOO order to NYSE, AMEX or Nasdaq automatically. I haven't had any issues with odd lots, as long as its over 100 shares. If its less than 100 then the order is usually rejected (at least for NYSE and AMEX, not sure about Nasdaq).
  • I haven't had any issues with orders being executed before the 9:30am open, and I'm kind of surprised as I thought that Pre-market orders had to be submitted as LMT.
  • hth,
  • -Rob
  • ________________________________
  • From: B S <bs2167@…>
  • To: TWSAPI@yahoogroups.com
  • Sent: Fri, May 28, 2010 10:32:25 AM
  • Subject: [TWS API] MOO Order Settings
  • I was hoping someone could help me with a couple novice questions on getting the correct order settings for submitting Market-On-Open orders. I have followed the instructions in the user guide and tried to get answers a few times via IB chat, but thus far have not been able to get a set of instructions that works for all cases. Essentially, all that I want to do is participate in the opening auction/cross for symbols that have either NYSE, NASDAQ, AMEX, ARCA in their TWS description. For example, I was told that for ARCA symbols I just needed to direct route to ARCA and use MKT and OPG — however, after selecting ARCA in TWS, the OPG option is not available in the TIF drop down. For NASDAQ, I was told to direct route to ISLAND and use MKT and OPG, but orders submitted that way executed immediately (prior to 9am EST) rather than waiting for the ~9:30 open.
  • If anyone would be willing to share a working set of order settings (routing, TIF, anything else relevant) for each of the 4 listings I would greatly appreciate it.
  • Also, if I might ask one additional question: assuming that these orders will be to initiate new positions, will the handling of odd lots/mixed lots be different than what IB has on their site for typical orders (i.e. direct routed odd lots will be rejected and the partial lot of a mixed lot will be canceled after the round lot is executed)? Thus far I have only tested round lots.
  • Thanks in advance for any input.
  • [Non-text portions of this message have been removed]
  • ————————————
  • Yahoo! Groups Links
  • [Non-text portions of this message have been removed]
  • Reply
  • rwk2095
  • May 28, 2010
  • View Source
  • — B S <bs2167@…> wrote:
  • > … will the handling of odd
  • >lots/mixed lots be different than what IB has on their site for
  • >typical orders (i.e. direct routed odd lots will be rejected and the
  • >partial lot of a mixed lot will be canceled after the round lot is
  • >executed)?
  • I frequently trade using MOO order on NYSE. I do not use MOO Orders
  • on Nasdaq because of liquidity constraints and would like to know if
  • anybody can share their experience with trading the opening cross. I
  • don't bother with AMEX because there is so little of interest there.
  • For NYSE and ARCA stocks, I specify NYSE as the exchange, MKT as the
  • order type, OPG as the time-in-force, and Smart as the routing.
  • Specifying anything other than Smart routing runs the cost up a lot.
  • I never had any trouble getting mixed lots filled, but NYSE doesn't
  • accept odd lots.
  • For Nasdaq stocks, I only use limit orders even when I am in a
  • hurry. Odd lots don't seem to be a problem.
  • [rwk]
  • Reply
  • B S
  • Message 4 of 4 , May 28, 2010
  • View Source
  • Rob & RWK –
  • Thanks very much for your input.  This should get me going in the right direction for Tues am.  Hopefully testing with these settings will go better than the last round.  If for some reason it still doesn't work, I will at least know that something other than my order settings is awry.  

[Q] How to update the future's margin requirement from TWS API?

[A] by christianhgross <christianhgross@…>

Do the following;

1) In the API track the account values including the margins, available

cash, etc, etc…

2) Create an order that is the inverse of the position that you are trying

to get the margin off. Thus if you are long 1 ES, create an order that is

short 1 ES.

3) Submit the order, but set the whatif flag to true.

4) In the resulting whatif callback you will be given a resulting

margin,cash report. Take those values and subtract them from the initial

account values and you have your margin report.

There is a catch in that if you are trying to margin complex trades that are

uncorrelated, or correlated you will need to "tweak" the calculation.

Experiment with this using the demo account.

Christian

[Q] How to get margin requirements for possible trade?

[A] by jb201448@yahoo.com

Added on 14-Apr-2016

[create what-if order…]

Hi Adam,

To my knowledge there haven't been any changes in the pre-trade margin information passed back through the API.

The only parameters I've ever really paid attention to are orderState.commission, orderState.initMargin, and orderState.maintMargin. These numbers match the "Post-trade Margin Impact" numbers available in Tws from performing an order preview. That is, they indicate the account's expected initial and maintenance margin values if the trade were to execute at the present time.

The number you are seeing ~1.79E308 likely corresponds to maxDouble- this is the API's way of saying 'null' for numbers.

Best,

-Josh

[Q] [TWS API] Stress-testing margin requirements using Risk Navigator to model changes in the underlying price?

[A] by grooooper maxr1r2@gmail.com via groups.io View/Reply Online (#38965)

Added on 18-Nov-2017

FWIW, after playing around with custom What-If scenarios in RN, and a fair amount of back and forth with IB, it appears that it's NOT possible to see potential margin impact of changes in the underlying prices, etc.  A frustrating limitation, since it seems rather straight-forward…all RN will do is model what changes in the underlying price will do to your P&L / overall portfolio value, but even though they show your account's current Margin / Excess Liquidity in RN, that figure does not update to reflect your hypothetical changes…

[Q] Figuring out Lookahead / Overnight Margin Rate For A Stock

[A] View/Reply Online (#38991)

I'm currently trying to ensure that my code does not violate the stricter overnight margin check starting at 3:50pm. I see how to check whether I will be in violation using reqAccountSummary with the lookahead tags. I also see how to check how a trade will affect my current margin by placing an order with WhatIf set to true. I do not however see a way to tell how an order will affect my margin in the future when the margin rates have changed, such as those used in calculating the lookahead values. It's seems important to know the future margin rate so that I can close as few positions as needed to pass the future margin check as waiting until 3:50pm when I could get the rates via API would put me at risk for margin call, unless I've misunderstood something.

I asked IB customer service if there was a way to get the future margin rates via API or some guidelines or rules I could follow to calculate them myself from the day margin rates, but IB customer services said there was not, so I thought I would try here just in case.  On IB's website they say the Stock Reg-T/overnight margin requirement for any security is 50%, but it's unclear if that's true for all securities or just a minimum.  I know the 25% initial margin rate for long stock positions reported on the same page is definitely a minimum and that IB often increases the margin rate for riskier securities.  Perhaps the overnight margin rate can just be calculated as max(50%, day maintenance margin rate)?  Does anyone have any experience to back this up?

How do others handle this problem?  I suppose I could just assume as conservative a margin rate as possible so that I error on the side of closing out more positions than I need, but I'd rather not.

 More

 Jack Jost Nov 27   #39012  

Hi,

IB publishes daily updates for rebate-/fee- and margin-rates via website and in CSV format on its FTP server as described in https://ibkr.info/article/2024. For margin rates format is in % of current price, volatile penny stocks can require to put up far more than 25% or 50%:

#SYM|CUR|NAME|CON|ISIN|CUSIP|LongMaintenanceMargin|LongInitialMargin|ShortMargin

[…]

SYAI|USD|SYSTEMS AMERICA INC|112161674|US8718773042|871877304|100|100|100

SYANY|USD|SYDBANK A-S-UNSPONSORED ADR|109785199|US87124G1022|87124G102|100|100|100

SYBF|USD|SYNTEC BIOFUEL INC|40429693|US87163W1099|87163W109|100|100|100

SYBRQ|USD|SYNERGY BRANDS INC|6858124|US87159E4026|87159E402|100|100|100

[…]

Short selling fees and availability is published in the same folder in this format:

#SYM|CUR|NAME|CON|ISIN|REBATERATE|FEERATE|AVAILABLE|

[…]

A|USD|AGILENT TECHNOLOGIES INC|1715006|XXXXXXXU1016|0.9100|0.2500|>10000000|

AA|USD|ALCOA CORP|251962528|XXXXXXX21065|0.7212|0.4388|5200000|

AAAP|USD|ADVANCED ACCELERATOR APP-ADR|212212690|XXXXXXXT1007|-7.3307|8.4907|100000|

AABA|USD|ALTABA INC|278946664|XXXXXXX61017|0.9100|0.2500|>10000000|

AABVF|USD|ABERDEEN INTERNATIONAL INC.|60152167|CA0030691012|-16.1002|17.2602|10000|

AAC|USD|AAC HOLDINGS INC|169041192|XXXXXXX71083|0.0439|1.1161|400000|

Hope that helps,

– J

 Adam S. Nov 29   #39026  

Jack, that's a great resource, although it doesn't seem to include information on the end of day margin rate.  That being said, it would be great initial and maintenance margin rates from one file instead of repeatedly querying the IB API one security at a time.  Do you know what the difference is between the multiple files that list the margin rates?  It seems like each file is labelled after a different country and the one ending in …-US.dat is missing US-listed securities (for example JNJ is not in it).  Oddly, the one ending in …-CAN.dat seems to have all US-listed securities although the margin rates seem a little off.  For example, JNJ is listed as having a margin rate of 30%, but in TWS it's 25%.

 Adam S. Nov 29   #39029  

Customer service referred me to someone in the trading group and they were able to answer my question.  The Reg-T/overnight margin check is separate from the other margin checks so the overnight margin rate is unrelated to the day's other margins and therefore the 50% is not a minimum but in fact true of all securities that are "marginable".  

 Jack Jost Dec 1   #39037  

The actual margin requirements depend on account type, in addition different IBKR entities (IB US, UK, CA, JP, HK, …) may have different margin requirements; and portfolio margin accounts obviously have different requirement than Reg-T margin accounts or cash accounts. Formulas and examples to compute initial, maintenance and overnight requirements are here: https://www.interactivebrokers.com/en/index.php?f=24862 (check the pull-out tabs marked with '+').

– J

[Q] Index of all available addons that can be purchased for tws?

[A] by Kurt

Did you look here?

http://individuals.interactivebrokers.com/en/general/poll/ibconsultants.php

[Q] Reverse engineer IB API socket protocol

[A1] by Ray Racine View/Reply Online (#45535) 

Added on 28-Sep-2020

I've been writing my own client library and the only and probably best reference I've found is the TWS library code.  However, it is sufficient.

The protocol is pretty simple.

Roughly:

1) There is only a tiny bit of "magic" bytes and protocol is in the logon / handshake.  After establishing the logon, the api message protocol is consistent and simple.  Request message encoding is found in the Client (EClient) file.  The response message decoding (the field layouts) are in the Decode (EDecode) file.  The call back method for a decoded message is in the "Wrapper" file.

2) For regular messages, nothing is binary encoded so no worries about net encoding, big endian, little endian ect.

3) A message packet is composed of null terminated strings or think of it as null delimited utf8 encoded string field values.

4) The prefix or header of a message is the length of the message followed by the message id (the type of message being sent) followed by the fields.

5) Some early messages have a version id as well, but latter messages do not.  The code for a specific request message indicates if a version is required as well as the version value.

6) Repeating fields, say a list of bars, are encoded as the number of elements in the list followed by the N repeating elements.

7) The protocol is async, you fire a request message to the server and depending on the request one of more response messages will ensue.

8) A request is associated with the one or more response messages (in general) via a Response Id or sometimes called a Ticker Id.  A naming without real semantic difference.  The only onus on a Request Id is that it should be a unique integer value.  The client synthesizes via counter, random generator or whatever, a unique integer value that is sent in the request (for those request messages that have a Request Id or Ticker Id)  and the server returns that value in the response messages so you can match responses to the specific request initiating it.

9) For inbound messages a thread is listening on the socket in an endless loop.  The loop consists of reading off the next messages byte length, reading in the N message bytes, parsing the message fields and then invoking the appropriate call back function for "message type" based on the message id integer value.

Once you dive in and do a couple of the smaller messages it is all repeat and rinse.

[A2] by Leno (elenosandoval)

(that one is hilarious)

Good Morning From The West Coast. Once in a while I try to contribute something to this great discussion group. I decided to stay flat this morning, so here's my piece for now.

Firstly, I'll try to clarify something by describing my own applications.

My personal program communicates with TWS using ASCII characters. It "Subscribes" to market data by sending a predefined set of characters to TCP port 7500, which is my TWS API "listen-and-talk-to-API-applications port." The default port is 7496, but I changed it to 7500 so that I will always know which is which.

Before I talk about which characters to send and which replies to listen for, let me say that TWS communicates with its servers over an encrypted link. At least I hope it does. I, personally, am not interested in the TWS-to-IB-Servers communications because, frankly, I would not want to know if my account info is vulnerable to hacking.

About how my program "talks" to TWS: No reverse engineering was necessary. All you have to do is look at the C++ sample code, for example, and you will find out which codes to send and how many and what kinds of "fields" to listen for in reply. There is sample code for C++, Visual Basic and Java.

I began my programming by first building a "receiver" before subscribing to anything, like market data, account data, order execution info, etc.

ASCII characters are exchanged between my program and TWS with a hexadecimal "00" as the field terminator.

For example, using the old serial port, I would send to comm. port 1, "Hello" followed by ASCII code 13 for "carriage return" than code 10 for "line Feed." That would tell the computer I'm talking to that that data transmission is ended.

The first "receivers" you'll need after logging in to TWS with your API, are the "Next Valid ID" and the TWS Messages receivers.

Here are the exact sequence of events that take place when my program logs into TWS:

1. Open a connection to port 7500 and send one field "20." That tells TWS that my program is at least up to version 20 in their scheme of things.

2. Read TWS reply version, which was 49 this morning. One field.

3. Read TWS's Date/Time string. One field.

That was the login sequence. Now my program goes into the main loop. Whenever there is a character waiting to be read at port 7500, my "receiver" reads the first field then it will know exactly what to expect next.

The first two receivers you'll need are the "next valid ID" code 9 and TWS Messages, code 4.

For example, read two fields containing "9 1" for code nine version one, then a third field containing my next valid ID for placing trade. It was 483 this morning. Wow! Time flys.

Here's what 2 TWS Messages looks like, code 4, followed by the version number, then 3 more fields:

4 2 -1 2106 HMDS data farm connection is OK:ushmds2a

4 2 0 165 Historical market Data Service query message: HMDS serveer connection was successful.

Okay, for now. I hope it helped out somebody.

Good luck,

Leno

[Q] Read this later for TAX purposes (if you ever need it at all)

https://groups.io/g/twsapi/topic/4045013

[Q] Can I get the MarketValue for all the currencies?

[A] by gamewworx

I was really looking for was not open positions but different market values in different currencies to calculate my risk.

I found out that you can get the MarketValue for all the currencies from UpdateAccountValue() method in EWrapper.

Steve.

[Q] Managed Accounts?

Hi all,

I used to run a hedge fund for which I traded with IB. Now I am looking at

running managed accounts instead. The big difference is that I would be trading

not on behalf of one entity (the hf) but many managed accounts. Currently I just

have one TWS associated with one account so API interfacing is obvious. How does

this work with, say, 20 managed accounts? I would need to loop throught the

account list, but with TWS and the card security check I do not see how that is

possible…Would anybody know of any documentation by the group or IB on the

matter? Because of how I trade I must computerize/ use API…

Any suggestions/hints much appreciated. Matt

[A] by Jens (from this msg)

This is the blog post:

http://leighdrogen.com/the-hedge-fund-structure-is-dead/

[A2] by Jens

According to IB, it is possible to setup a structure so you trade in one

master account, and all trades you do are automatically split out to all

your managed accounts. Leigh Drogen at Surfview capital wrote a blog

about this once – http://leighdrogen.com/ you can always check with

him, I think he may be using IB as well. In any case, I am sure it can

be done quite "easily" with the IB platform.

Good luck!

Jens

[A3] by Mal Spink

You can set up an allocation profile and specify the percentage of shares executed to be allocated to each account.

[A4] by Matthew

I have been trading client accounts in a similar configuration to that which you have explained for many years now. On the most part it is quite functional and achievable.

Some of the things I learned (but not documented very well)

* There are a number of ways to raised orders to FA type accounts, (i) using an FA profile (which needs to be setup in TWS first), (ii) as a general order which is allocated according to a set of rules in using an FA method setup in the API application or (iii) allocated to a single account. For the clients I serve I use all 3 combinations.

It is important that the IB account is setup correctly to receive each of these types of orders.

* Portfolio events have not always been reliable on sub-accounts. I found the data did not always come through and the only way I could force it to refresh was to request the portfolio details for the parent account and then re-request the portfolio details for the sub-account. Something to be on the look out for.

These were probably the key pieces of information worth sharing, I hope they are of use to you … good luck.

Regards,

Matt S

[Q] Getting a full list of Exchange Listings via API?

[A] by Josh View/Reply Online (#46638)

Added on 26-Feb-2021

Unfortunately no there’s nothing like that from the API. The website should be up to date with available exchanges, there are quite a few on there.

Update: There are 253 now on IBKR. You can request this from API using reqMktDepthExchanges().

https://interactivebrokers.github.io/tws-api/market_depth.html#reqmktdepthexchanges

– Bruce


Reporting

[Q] How Do I Match the OrdeId that is sent with the Executed Order ID that appears in the Trades and Summary tabs?

[A] by Kurt (from this thread)

I don't remember all the details about the TWS Trades window itself.

However as an API programmer you should not need that info. You can use

info from reqExecutions which in addition to the execId field (which maybe

matches the field displayed in TWS), includes the permId, clientId, and

orderId fields which you can match with your API-submitted order info.

However as I recall there is no supported way to extract complete "drill

down" information from the Execution struct info to help you associate

related events particularly in the case of combo orders. It can get pretty

complex if you have combinations of any and all of:

split executions

orders split between accounts in the FA case

multi-leg orders

So as I recall the drill-down displayed in TWS can not be fully recovered.

But note that you *do* get execution reports both for the individual legs

and for the combo itself.

Also last I checked there is no way to associate an API order with orders on

downloaded activity reports. Yet another execution id is used there. As I

recall there is nothing to do except to match up contract, size, and time.

-Kurt

[Q] save executions

[A] by Kurt (from this thread)

> Hi,

> I see that occasionally I am not notified on an execution of an order. I get

> this intermittent and haven't found any connection between the cases.

> Has anyone encountered such a problem?

> I could do without the executions because From time to time I call

> reqExecutions(). When I call reqExecutions() I would like to get all the

> executions from the last week, in case I missed any one. The problem is that

> the executions I get depend on the checkbox of TWS trades window, which is

> usually checked only for today. Is there a programmatic way (even by

> changing a configuration file) to change these checkboxes from my

> application? In this way I will check all the days and then return it to the

> current day automatically.

> Thanks,

> Nir

You will have to set the checkboxes by hand.

Note that if you use IB Gateway you will not be given the choice and will

only get today's trades.

The best way to achieve reliability is to make sure you collect the info

every day, or to use one of the other mechanisms, the executions.txt file or

the "Exported" executions files which are saved *indefinitely* (no 1-week

limit). For your reference see my previous message about that below.

-Kurt

—— Forwarded Message

From: Kurt Bigler <kkb@…>

Date: Tue, 25 Jan 2011 01:01:53 -0700

To: "TWSAPI@yahoogroups.com" <TWSAPI@yahoogroups.com>

Conversation: [TWS API] Re: Get execution example Java

Subject: Re: [TWS API] Re: Get execution example Java

Auto Export (under TWS config) lets you specify how often you want execution

info exported (to a directory of your choice). You can also specify columns

to be included. I noticed that when I changed the columns in the export the

columns in executions.txt changed similarly.

So I would guess that the Auto Export "Interval" might also affect how often

executions.txt gets written out, but I did not confirm. In any case the

interval is specified in minutes, so I would not expect sub-minute updates

unless exeuctions.txt is handled differently.

I believe I had to use "Extended Form" to get sufficient information to

identify trades for my purposes. (I thought I also tried Custom Columns

once, but it is not set now, so I hope Extended Form isn't missing

anything.)

-Kurt

[Q] orderStatus function is be called 3 or 4 times(!) even when there is no update of status. Wtf?!

[A] read this whole thread

[Q] How to track order fills?

[A] There are FOUR ways we know of to get fill information.. read this whole thread

[Q] How to obtain the complete current portfolio?

Hello:

I am trying to export the current portfolio for an account to a csv file automatically using C++ code to the IB Traders Gateway API.

The reqAccountUpdates(bool subscribe, const IBString & acctCode) seems to only to provide updates to the portfolio, should they occur in the future, if I understand the API C++ reference manual correctly.

How can I obtain the whole current portfolio instead?

– Randall

[A] by rwk was found in this thread

Added on 05-Jan-2016

Maybe updatePortfolio() will give you what you need.  Call reqAccountUpdates() and you get everything at that point, plus you get all changes as they occur.

[rwk]

——————————————-

(confirmed)

This works after carefully working with the contract fields. I even added in a total P/L cell in the csv spreadsheet

At 04:12 PM 10/21/2015, you wrote:

Thank you for the reply.

I do need to point out the following:

The following header and 1 data line is from the "export the portfolio" menu command on the TWS desktop:

Contract Description,Exchange,Position,Currency,Market Price,Market Value,Average Price,Unrealized P&L,Realized P&L,Liquidate Last,Security Type

Portfolio,,,,,,,,,,

WMT,NYSE,100,USD,58.6550026,5865.50,58.72,-6.50,0.00,No,STK

The suggestion to use the C++ API call reqPositions() request to obtain the current portfolio, results in the following:

Account,Symbol,Shares,Price

ZZ000000,WMT,100,58.72

How do I obtain all the missing fields in the first entry above?  More than a reqPositions() call is necessary, obviously.

The current Position in the portfolio export is the same as the current position in the reqPositions() call?  I guess it will be necessary to save the avg price upon purchase of the stock?

– Randall

[Q] Is there a way I can get total short value of my account via the API? I'm basically looking for the total value of all short positions.

[A] by Kurt Bigler View/Reply Online (#38959)

Added on 15-Nov-2017

Take a look at all the information that comes back from the account and position callbacks (updateAccountValue and updatePortfolio) when you subscribe to that info.  Look at it and you will see exactly what's there.  It is pretty transparent for the most, as long as you understand the notions around currencies and base currency.

The information you receive through those interfaces is basically exactly the same as what you get in TWS from the position columns (particularly under the Portfolio tab) and from the Account Window.  Don't consider using the API without using TWS, since TWS is in effect the living documentation of the API, and you will waste a lot of time having questions about what is made obvious via TWS.  TWS is generally a superset, but if you don't see something in TWS you won't find it in the API.

As I recall off-hand I don't believe that TWS (or the API) separates out total long from total short, as you might find in the web-based account management reports.  You will probably have to add that up yourself.  As your positions are enumerated by the API, keep a running updated table of your positions by contract, with associated currency or base currency value, and iterate through all you positions to obtain total long or total short, or (more efficiently perhaps) maintain a running total long and total short (perhaps by currency) which you can update whenever a position value changes by accruing new position values and de-accruing the prior position value.  I'm hoping that makes sense.  I'm trying to refer to a fairly basic technique in which an incoming value is processed in relation to the previous stored value for the same (if any) and global dependent values are updated accordingly.

Once you get total long and total short you can combined them and combine with cash and compare against your net liquidation value as a double-check.  Account values may always have slight discrepancies in relation to position-based valuations, and are considered more authoritative than what is computed from position.  You can see the same in TWS if you set up total rows to double-check against Account Window values.

You man want to process accountDownloadEnd to know when you have received a consistent snapshot of your entire account picture.

-Kurt

[Q] Reported realizedPNL got reset in the random time during the trading hours, how to change that time?

During the middle of the trading session, the realizedPnL got reset, does anyone know on top of your head where on the TWS settings to set when to reset realizePnL, so I can adjust the time non trading time.

Thanks

jeff

[A] by Josh, View/Reply Online (#41522)

Added on 07-Jul-2020

Jeff,

The realizedPnL for the daily PnL functions reqPnL, reqPnLSingle is reset by default on an instrument-specific schedule, but can be modified through Global Config settings at General -> PnL Time Zone. The realizedPnL returned from reqAccountUpdates and corresponding to values in the TWS Account Window is reset once per day but not at a configurable set time.

Josh

[Q] How to get PnL of Trades?

Is there any easy way of getting this information in API?

I am interested in PnL of the trade.

I am aware of reqPnL and reqPnLSingle, my understanding is they give the realized and unrealized PnL. I am interested in realized PnL broken up by trades.

[A] View/Reply Online (#44090)

Added on 18-Oct-2020

The realized PnL for your closed trades you get via the commission report. This event is triggered along with the execution details event.

When requesting executions you will receive all executions + commission reports for the last 24h.

You can get up to 7 days by altering the setting in trade log in the TWS workstation

>I've read elsewhere that you also need a gateway for the rest api.

Is that also correct?

Yes, correct. Read this:

https://groups.io/g/twsapi/topic/72607088#44154

>Having 90 days of executions would help a lot.

Yes, would be fine. Unfortunately it is not provided when using IB API even if this would technically also be possible. I think because IB has to find at least one competitive reason to use the Rest API against IB API. Such marketing decision is not very courteous at all.

If you only need execution info – just take a look at flex reports. As far as I know data can go as far as 5 years back (reports are also limited to 5 years)

The problem with flex reports(even though I use them) is that you cannot directly map them to an order. They do not link to the PERMID or ORDERID. It's anoter id. God knows why. At least the execution ids are the same.

Had to come up with a formula to reconciliate the orders.

— abit later: "The Formula" —-

Hello,

Yes, I have a query that helps me reconcile the orders.

I use C# for the query so the query will be in that language:

flexOrder.Symbol == order.Symbol

&& (flexOrder.Quantity == (flexOrder.BuySell == Enums.BuySell.BUY ? 1 : -1) * order.NrOfShares)

&& flexOrder.BuySell?.ToString() == order.Action

&& flexOrderCreationUtcTime.AddHours(-1) <= order.CreatedDate && order.CreatedDate <= flexOrderCreationUtcTime.AddHours(4)

As a note, I save my orders in UTC date time, so I have to bring the flex query order from NewYork Time to UTC.

For IB timezones please take a llok at this topic

https://groups.io/g/twsapi/topic/69936034#44206

If you're using c# you canm get a headstart by using this library:

https://github.com/gabbersepp/ib-flex-reader

Best Regards,

Dragos Durlut


Not yet confirmed (but possible issues)

[Q] Validity of Paper Account for Testing Algo Trading Outcomes?

[A] by tdrtw (found here)

Based off my experience, it's a no-no.

I've found my paper account will execute at prices that never happened nor are on the charts. I use the paper account only for testing functionality of code, not expected values of a trading system.

Cheers,

Damien

[A2] by Pankaj Sharma  (found here)

One reason results differ is that a paper account evaluates order fills only

when the traded price changes.

Pankaj

[Q] Other limits I’d be aware of?

[A] Order limit ~50 per second

[Q] there is no correlation between bid/ask and trades in IB's feed (?!!)

[A] by Ed news1000@edshome.net

I don't know what data the exchanges publish but I do know that there is no correlation between bid/ask and trades in IB's feed.

Bid/ask and ticks are independent streams and there is no way to determine what the bid/ask was when you receive a tick. In addition, sometimes you the see the bid/ask change before the tick arrives, sometimes you see bid/ask change after the tick arrives.

If someone has solved the puzzle and can get accurate trades at bid/ask from IB I would be happy to hear it.

I suppose I should state explicitly that this is not a rant or complaint. It is simply my understanding of how IB's realtime data works.

[A2] by rwk

@Ed,  You're correct that there is no connection between the order book and the tape, but this is the same for all brokers, not just IB.  Some data vendors transmit the entire price ladder as one transaction.  While that makes programming simpler, I don't see how it reveals any more information.  It just uses up bandwidth.

When an order arrives at the exchange, it is either immediately marketable or not.  If marketable, it is matched with a an order from the book and printed to the tape.  If not marketable, it's added to the book at the end of the queue for that price.  The order book contains only orders, no trades.  The tape has only trades, no orders.

You really need both the book and tape to see what's going on these days.  In the 19th and early 20th centuries, only the specialist could see the book, so the tape was all we had.  There is a substantial collection of folklore about how to read and interpret the tape.  But we're into the 21st century now, and everything happens fast … VERY fast.  I'm not sure that old wisdom has any relevance now.  At any rate, I am not smart enough to build a profitable system from the book and tape, and faster data would likely not help much.

[rwk]

[Q] updatePortfolioEx() non consistent behavior?

[A] by "hlhhtl" <harry.webmail@…>

Okay, as no one seems to have the same issues or wants to comment on it I will share what I have done meanwhile. Maybe live trading members can add their experience with updatePortfolioEx():

I changed from simultaneously subscribing (reqAccountUpdates()) to all the particular accounts to iterating, meaning I subscribe for a single account once, (so far) wait 10 sec, unsubscribe wait 5 sec, and then do the same for the next account.

This seems to give me more stable and correct data (at least for my very first tests). Btw, this is true for FA paper as well as live accounts.

I guess I will change the timed request with a combination of time and AccountDownloadEndUpdate() event, but still it leaves me wondering what I could do better or what is wrong with the API. Of course, IB did not deem it necessary at all to answer my questions so far.

It would be nice – and very basic, to my understanding – having a broker API which reliably and contniously gives you your account equity (not just 0 on every second request) and the positions held in all your FA subaccounts.

— In TWSAPI@yahoogroups.com, "hlhhtl" <harry.webmail@…> wrote:

>

> Hi,

>

> I am subscribing to several FA sub accounts (mostly a group of three) under VB.NET with ActiveX reqAccountUpdates() method.

>

> I read data with updatePortfolioEx() but I do not get the correct data back. Despite waiting for and getting accountDownloadEnd() and correct data for the accounts, the portfolio (=positions) data is incomplete, wrong, then (partially) correct, then incorrectly showing 0 again for positions,etc. Also, not all accounts get updated: sometimes only one or two out of three or more accounts show any data at all, but even if there is some data for all the accounts, it is incomplete or changes from correct to correct back and fort as described above.

> Any advice? Thank you!

>

[Q] Access multiple accounts?

[A] by Kurt

From your original message I was not clear that you were attempting to

subscribe to several subaccounts at once. The API does not permit that, as

it seems you have figured out. Yes this is a major bug as far as I am

concerned and a real PITA, a cause of much coding shenanigans to produce

absurd workarounds for the seemingly idiotic limitation.

FYI the account you subscribe to switches the current account in TWS, in

case you didn't notice. And there is only one current account in TWS,

although in TWS (but not the API) you can also select an account group and

get the totals.

The only good side of how it if I am not mistaken is is that you do at least

get updates immediately (or without too long a delay) when a position

changes (due to an execution).

And the rest of the time you can multiply your position by the live market

data to determine your position value (and subtract the cost basis to get

unrealized P&L). This gives you faster updates rather than relying on the

periodic reqAccountUpdates info for the position value and unrealized P&L.

One strategy that might work is to subscribe to the "All" account which is

obtained by using the master account name with "A" appended. Then when

activity comes in (e.g. position change) on the All account you know to

rotate through the other accounts to get the details. That is not a 100%

solution but pretty close and might let you avoid constantly cycling, which

also interferes with TWS operation, if you use it.

I was going to say please add your vote to my request for this to be

improved in the feature poll but apparently all my submissions to the

feature poll have disappeared, and I am on the phone to IB right now about

that.

In the meantime please submit a feature poll request and notify us here of

the poll item number or URL here so we can vote on it.

-Kurt

[Q] How to monitor events (keyboard, mouse) in X?

Looks like my remote VNC session didn’t pass keypress events properly to remote (update: actually it was just due to TWS GUI was in “Locked” state – new feature from IB:)

[A] found here:

http://unix.stackexchange.com/questions/146287/monitoring-events-keyboard-mouse-in-x

(quote)

A pretty quick search would likely yield xev as a result. It will not show you everything that is pressed or typed in X ever. But rather, will allow you to see information about keycodes and mouse movements.

However, with the -root option, you might be able to get xev to monitor the whole X session. Note, if you do this, you'll make it pretty difficult to actually interact with any program since xevwill be capturing all input.

xev -root comes with an implied "YMMV". Another possible solution is to use xinput to get the job done (though you need a recent version (1.6.1 or above)):

$ xinput test-xi2 –root

While the above should be able to capture all events, if your xinput doesn't support test-xi2 or–root, you can use xinput test on a particular keyboard or mouse device.

See also this other question: Record every keystroke and store in a file

(end of quote)


APPENDIX

[GOOD IDEA]

Since with the "snapshot" parameter turned on you only get a relatively small set of data (and quite fast) you can circle around many symbols (certainly much more than 100) by requesting snapshots in a round-buffer manner.

Also good idea:

I use a Producer/Consumer async queue, so each income message is pushed to the queue, that's it, the reader thread doesn't wait for the real handling to finish.

You can have multiple queues, for example for each symbol (or "subscriber" or whatever). Of course, like in economics, the law of diminishing marginal returns rules, so don't overload with consumer threads…

Yet another one:)

VolumeSense is a charting application for Interactive Brokers focused on volume and volume action. It creates intuitive charts displaying the volume traded at the bid versus the volume traded at the ask for any traded price.

[contract id is the only reliable ground out there]

I manually create all my option contracts when they are initially entered

via UI. In your example I type AIG:jul40c and the symbol, expiry (probably

gets converted to "201007"), strike, and right fields are set from that.

Calling request contract details with that I get a more complete contract

back from which I update some of my Contract fields, maybe even all of them.

But I definitely stash the contract id and save it for use when the

workspace is reloaded, since that allows the symbol to change overnight

without consequence.

— Kurt

[todo]

check out liquidity of this tool:

http://en.wikipedia.org/wiki/E-mini_S%26P

[other implementation notes]

===============================================

> However, (as was pointed out by others), there's no actual

> reader thread. Is that really true, or did I miss something.

Yes, thank god. Like mr mole, I am procedural.

The reader thread is missing perhaps because C++ does not provide the nice

transparent synchronization stuff that java provides.

Actually I *still* had to do some extra stuff to get it all working,

specifically *avoid* any blocking so I can run the UI and other kinds of

connections from the main thread. Ever so slightly inefficient but a not

measurably so for my use, and it results in a damn simple model with no need

to even *think* about synchronization.

Anyway for that purposes I had to subclass EPosixClientSocket and override

checkMessages, and indeed call select().

Here is the code in case anyone has use for it. There are a couple tricky

cases to handle to make it possible to do this by subclassing.

EPosixClientSocket was a bit unfriendly in terms of making it less than a

pain to achieve nonblocking behavior without hacking up EPosixClientSocket

itself.

class SocketClient : public EPosixClientSocket {

private:

typedef EPosixClientSocket inherited;

public:

SocketClient( MyWrapper *ptr);

MyWrapper *GetWrapper();

bool checkMessages();

bool hasDataToReceive ();

};

SocketClient::SocketClient( MyWrapper *ptr)

: EPosixClientSocket (ptr)

{

}

MyWrapper *SocketClient::GetWrapper()

{

return static_cast<MyWrapper *> (getWrapper()); // constructor argument

type assures this static_cast is valid

}

bool SocketClient::checkMessages()

{

if (isConnected ())

{

if (!hasDataToReceive())

return false;

else

{

// Replicate this test from top of (inherited)

EPosixClientSocket::checkMessages.

// Eliminating this possibility (isSocketOK false) assures bool

result below comes from bufferedRead, thus from receive, thus from ::recv.

if( !isSocketOK())

return false;

bool result = inherited::checkMessages();

if (!result)

{

eDisconnect();

getWrapper()->connectionClosed();

}

return result;

}

}

else

return inherited::checkMessages();

}

bool SocketClient::hasDataToReceive ()

{

// Calling ::recv with MSG_PEEK might conceivably be more efficient than

using select for a single socket.

int sock = fd();

fd_set socks; // but only one

FD_ZERO (&socks);

FD_SET (sock, &socks);

struct timeval timeout;

timeout.tv_sec = 0;

timeout.tv_usec = 0;

int readsocks = select (sock+1, &socks, (fd_set *) 0, (fd_set *) 0,

&timeout);

return readsocks > 0;

}

-Kurt

===============================================

If an API app requests the same market data N times the ticker will get

counted N times, or at least that is what IB told me on the phone. I coded

my API app to assure that all "internal" subscribers to the same ticker

produce only one "external" (API) subscription.

-Kurt

===============================================

[consider using 2 TWS (paper/real) same time]

by Richard L King rlking@aultan.com

In fact you CAN use the live data simultaneously on both the paper and the real account, provided that both TWS instances are running on the same computer. I do it all day long every day, and have done since the paper trading system was introduced. (It used to be the case that the two TWSs could be on different computers, but IB closed off that option a few years ago, apparently after complaints from the exchanges.)

I hope the second sentence in your quote isn’t an indication that IB are about to restrict it further. What use would the paper trading account be if you couldn’t make effective use of it when the live account is running?

===============================================

[important: Kurt to Jan, about TwsApiC++]

Kurt Bigler

Jul 30, 2010

View Source

Jan,

I thought you were no longer around, since you had not posted for maybe a

couple years, a fact which I mentioned on this list a couple times.

Several months ago I divested myself from TwsApiC++, given the availability

of POSIX support, and what looked like an extremely difficult process to

move the TwsApiC++ code to 9.64. I already had TwsApiC++ running fine with

9.60 for I think almost a year. I did have to add a buffer layer to the

TwsApiC++ implementation in order to achieve single-thread operation as

required for my app.

With the 9.64 POSIX TWS API I had one remaining problem: I had to subclass

EPosixClientSocket in order to achieve non-blocking reads from the socket,

so I could run the UI and the socket communication both from the main

thread. I posted some code in connection to this on 21-Jul in the thread:

Re: [TWS API] Sigh… I hate windows (and I hate TWSAPI)….. threading

I also have a lot of notes that I intended to pass on to you, related to

issues I ran into in getting TwsApiC++ working. I never managed to get them

to you when you seemed to disappear, but I'm not sure how relevant they are

at this point. I think the notes pertained to the 9.60 effort but I'm not

sure.

I should also mention that in my case the target platform is the Mac.

-Kurt

and Jan’s reply goes like this:

JanB

Message 3 of 3 , Aug 5, 2010

View Source

Hi Kurt,

I have been around, but more as a silent passive user towards this group I must admit. Although, I'm still using TWS and it's C++ api. And as explained in my post and from some reaction via email after posting the 9.60 version, it looks like help with the C++ api is still welcome and appreciated., so I got re-motivated.

As you pointed out, moving the 'pre-posix' TwsApiC++ implementation into the new 'post-posix' era of the tws c++ api, is quite complex. Although having a nearly finished version available, I think it is the wrong approach with the new posix stuff available. Using the posix interface avoids the need to hack the afx… stuff and it gives greater freedom to hook the api because of the extra methods in EPosixClientSocket. Moving the 'pre-posix' twsapic++ code helped me to understand it better, and while slowly getting convinced of leaving behind the 'old' stuff, your code example gave it the last needed push to do so. So Kurt, thanks for pointing me to your code!

This all leads me to redefine the purpose of the TwsApiC++ project:

Provide the TWS C++ user with the same interface as the pre-posix implementation by hiding all the socket stuff and, of course, it should still be platform independant and portable.

This will materialise in TwsApiC++ version 9.63 as follows:

a) it supports basicly the same interface as previous versions, unlike the posix C++ Api;

b) Use the IBString instead of the TwsApiC++ specific xstring

c) it will support both i) a synchronous non-blocking call to checkMessages, and ii) an automatic,asynchronous reading of the incoming events in a seperate thread. Two new methods are added for this: checkMessagesStart() and checkMessagesStop(). When started the thread, the synchronous checkMessages() becomes void (does nothing).

d) TwsApiDefs.h will be reworked so that the strings and the functions are implemented only once to avoid linking problems, plus one line will define the enumeration and the strings, i.e.

what used to be

EnumsOf( FormatDate ) { FD_AsDate=1, FD_AsSecondsSince };

Strings( FormatDate ) { "AsDate" , "AsSecondsSince"};

EnumsAndStrings( FormatDate, FD_AsDate, FD_AsSecondsSince )

becomes

TWSValues( FormatDate )

{ TWS_V(FD_,AsDate , 1 )        // starts with value = 1

, TWS_N(FD_,AsSecondsSince )

};

TWSFunctions( FormatDate, TWS_N(FD_,AsDate), TWS_N(FD_,AsSecondsSince) )

When the string values are different, then:

// Execption on the rule on the naming convention.

TWSValues( Rule80A )

{ TWS_S(R8_, Individual , "I" )

e) On windows, the MVC Studio 6 is dropped in favour for the MVC C++ Express Edition 2008.

Kurt, any note you think might be relevant, I would be pleased to learn about. Also, having invested in pushing all the events on a queue ready to be processed is still of value I believe, especially in the current multi core processors era. Reading and parsing the socket output should be considered time consuming, so when that can be done in another thread and possible another core than the one used by the main processing thread in your ATS, just the better. It is like reading/writing a file in a separate thread in your program.

Kind regards,

Jan

[Q] Good alternatives to IB?

[A]

From this thread: View/Reply Online (#47914)

I just want to double check if the information i found is correct for TickByTick data. By default user has 100 market data lines and each tbt data for one ticker is worth 100 lines. You get minimum 3 tickers, which is essentially equivalent of 300 lines.

If you would want 4 tickers tho, you would need 400 lines, but that would mean buying 3 booster packs, which is $90 for 4 tickers using tbt. Meaning, if you want 10 tickers using tbt, you need to get 9 booster packs, which is $270 per month for 10 tickers? With hard limit being 10 booster packs, which means the max you can have is 11 tickers using tbt.

Let me please know if i got it right.

If that is the case, then it seems obvious something like Polygion.io, which is offering i believe unlimited tickers for $200 is a better deal.

I believe your calculation is correct but you can also get "free" data lines for a sufficiently active or large account. Check out the impact of monthly commissions and portfolio value under "How Market Data is Allocated" at https://www.interactivebrokers.com/en/index.php?f=14193

There may be other reasons to use a different real-time data provider. For example, when lag or choppiness of the data are important, when brief interruptions during the maintenance window cause issues, or when time stamps with resolutions of less than one second are required. Keep in mind that IB is not a data provider and their infrastructure is optimized to support human-centric trading via TWS and trade automation via the TWS API.

It looks like polygon.io an interesting offering for US stocks but they don't have futures (yet?) that we are more interested in.

JR

——————————- Also:

I use the Nanex data feed for pricing. Those guys are really sharp, and

the datafeed (while $$$$) is unbelievably awesome. Full level I for

all U.S. equities and all options, averages about 2Mb/sec during the

day (bursts to >20Mb on open, close and FOMC announcements).

Uses <2% of my CPU. Simply astonishing.

[implementation notes]

My code would be of no use to you since it is integrated with an extensive

Macintosh UI and I have quite a few layers of my own abstractions between my

request for market data and the underlying TWS API functions.

— Kurt

[there was an intention/attempt to build TWS API wiki / stackoverflow, but…]

Adam Hardy

Message 3 of 4 , Aug 25, 2010

View Source

My 2 cents: setting up categories is a low success venture and to increase the

probability that it works you probably need someone who is prepared to dedicate

the time to 'gardening' the FAQ (or wiki or forum), which will result in the

evolution of the best fitting sections, groupings or categories of subject.

i.e. start off very basic and manage it as it grows.I think the topics you

listed in your last paragraph would make a good start.


Time drift

                                                        latte-art-dali-clock.jpg

What time is it?

The Time is Now! Or more accurately, we are temporally balanced on the infinitesimally small edge between what will be and what was. For as soon as you say "Now", your word is in the past and can no longer be accurately called "Now". Further, with this in mind and actually within our minds, as our brains take some amount of time to process the "Now", we technically, and continuously live in the past; existing in the fractional fall-off from the true peak of the wave between what will be and what was.

"Time, time, time, look what's become of me...."

MarketMole  

While collecting data and timestamping it with local time Dmitry noticed nasty time drift up to few seconds a day. Syncing local time only once a day before breakfast appeared to be far from enough to keep things in order. The situation with time was not acceptable and was out of control so after googling around Dmitry found out that NTP is the best solution out there (works for all OSes). For windows use Meinberg NTP.

You should expect a time error in +/- 15..30 milliseconds range when polling time updates from public NTP servers.

To help NTP protocol to sync your boxes even better you can consider building your own Stratum-1 NTP server on your LAN (less than $100 on parts), which would decrease time error on all your boxes polling time from it down to +/- 2,5 milliseconds!

[Q] Does my/your/any computer have a time drift?

[A] Yes 🙂

[Q] Should I build my own NTP server (on my LAN) or would it be sufficient to just use public ones?

[A] Depends on the level of time accuracy you want to achieve:

  • If having +/- 15..30 millisec is ok with you, then just install NTP (for windows use Meinberg NTP)

  • If you want up to +/- 2 millisecond precision, you’d rather build own NTP server and poll updates from it as often as NTP configuration allows you (each 8 seconds). This is how to build own NTP 🙂

  • To further improve time precision you’d convert your computer into NTP server itself, add GPS PPS impulses on some serial port or buy special card with ext. antena jack, then you would have time sync “polls” every second (with ~ 4 microsec precision), but I’m not going to implement this. +/- 2 millisec is good enough considering we have network jitter 1 order of magnitude larger.

[Q] How to build own Stratum-1 NTP server on Raspberry Pi?

[A] Dmitry's notes: How to build own Stratum-1 NTP server on Raspberry-Pi + GPS (PPS)

A man with one watch has an illusion; with two comes confusion; with three an error requires collusion.


Some unsorted yet related stuff

[about the choices we do…]

I know, but I believe it is still less trouble per person long-term than it

is to stay at yahoo.

We also decline the position of power by staying, and perhaps that is the

single most common mistake people make nowadays. (It *is* in fact a bad

trading habit as well.) People stay with things that are not worth staying

with and as a result many things do not improve. The cost of that choice is

very high.

Sometimes choices are clearer when you see them as examples of something

more fundamental. Choices made that way are in my experience very

satisfying, and give me a great not-looking-back feeling. A sense of raison

d'être, if you will.

-Kurt

=================================================================

Look into the accumulate/distribute algo (and other?).

http://www.interactivebrokers.com/en/trading/orders/accumulatedistribute.php?ib_entity=inst

how to fly: https://groups.io/g/twsapi/message/23348

🙂

=================================================================

For a market that has a central limit order book such as ES, you should expect to pay one tick each for buying and selling.

=================================================================

by rwk:

ES is very popular because of price movement and liquidity. But it is also very efficient and hard to beat consistently.

My experience has been that the low-hanging fruit is more often found in stocks. But compared with futures, stocks have many disadvantages: much harder to short, less leverage, fewer tax advantages, and no central limit order book.

=================================================================

Kurt:

 I wanted to evaluate lags between the different streams.  

So consider LAST price as an example, or it probably works with any other price data.  Based on the incoming sampled data, even though it is sampled, you have as a result a price which is a continuous function of time.  That function is updated with each incoming sample but I think it is important to think of it as continuous.  In generating pseudo-continuous data that can be compared across streams it is important to assure that the time references are exactly the same.  My memory is that there is no guarantee of this and you have to make sure.  This is why I suggested that when you collect time stamps you collect the time the data was actually read from the socket.  LastTimestamp as I recall does not come from the exchange, but I think actually comes from the local TWS or Gateway server.  The TWS or Gateway server has a timestamp that can float in various ways and will attempt to correct itself in various situations, I seem to recall, in relation to the host's local time facility.  There was a discussion of this in considerable detail.

….

First thought is if you simply do a mean-square-error calculation comparing these functions ofcontinuous time, then you can get a certain estimate of error between any two streams.  This does not necessarily tell you anything directly about lag (or the direction of it).  It is possible to develop a statistic that would tend to measure lag more specifically but that is really only important if the errors show the need for it.  Once you have the ability to measure an error between any pair of streams you can apply it to all possible pairs, and make sure it is "agnostic" with respsect to the types of account (regular vs paper) involved in the pair.  If it is NOT, then there is perhaps reason to develop statistics related to lag.  I suspect it will not be necessary, and we just want to prove that through the consistency of the error statistics across all pairs.

    global m;

    global m1;

    global m2;

    global m3;

    m =   importdata('d:/workbench/20140501_port_7496_ABX.parsed.continuous');

    m1 = importdata('d:/workbench/20140501_port_17496_ABX.parsed.continuous');

    m2 = importdata('d:/workbench/20140501_port_27496_ABX.parsed.continuous');

    m3 = importdata('d:/workbench/20140501_port_37496_ABX.parsed.continuous');

    ping_google = importdata('d:/workbench/_20140501004701_ping_google.com.txt.parsed.continuous.x');

    ping_gw = importdata('d:/workbench/_20140501004621_ping_206.53.61.227.txt.parsed.continuous.x');

   

    global x y;

    global x1 y1;

    global x2 y2;

    global x3 y3;

    % extract (x,y) from 4 datasets

    x = m(:,1); y = m(:,2);

    x1 = m1(:,1); y1 = m1(:,2);

    x2 = m2(:,1); y2 = m2(:,2);

    x3 = m3(:,1); y3 = m3(:,2);

plot(x,y)

plot(x,y-y1)

plot(x,(y-y1).^2)

>> sum((y-y1).^2)

ans =

    1.5746

>> length(x)

ans =

      107995

>> sum((y-y1).^2) / (length(x)-1)

ans =

   1.4580e-05

>>

>> sqrt( sum((y-y1).^2) / (length(x)-1) )

ans =

    0.0038 ($)

Acc1_real   Acc1_paper

Acc2_real   Acc2_paper

[Q] Which pairs we’ll compare?

[A] For each ticker we’ll compare all combinations of 4 streams (6 pairs):

r1_p1

r1_p2

r1_r2

r2_p1

r2_p2

P1_p2