recentpopularlog in

jerryking : traffic_analysis   7

Spy tactics can spot consumer trends
MARCH 22, 2016 | Financial Times | John Reed.
Israel’s military spies are skilled at sifting through large amounts of information — emails, phone calls, location data — to find the proverbial needle in a haystack: a suspicious event or anomalous pattern that could be the warning of a security threat.....So it is no surprise that many companies ask Israeli start-ups for help in data analysis. The start-ups, often founded by former military intelligence officers, are using the methods of crunching data deployed in spycraft to help commercial clients. These might range from businesses tracking customer behaviour to financial institutions trying to root out online fraud......Mamram is the Israel Defense Forces’ elite computing unit.
analytics  consumer_behavior  cyber_security  data  e-mail  haystacks  hedge_funds  IDF  insights  intelligence_analysts  Israel  Israeli  Mamram  maritime  massive_data_sets  security_&_intelligence  shipping  spycraft  start_ups  tracking  traffic_analysis  trends  trend_spotting 
april 2019 by jerryking
Algos know more about us than we do about ourselves
NOVEMBER 24, 2017 | Financial Time | John Dizard.

When intelligence collectors and analysts take an interest in you, they usually start not by monitoring the content of your calls or messages, but by looking at the patterns of your communications. Who are you calling, how often and in what sequence? What topics do you comment on in social media?

This is called traffic analysis, and it can give a pretty good notion of what you and the people you know are thinking and what you are preparing to do. Traffic analysis started as a military intelligence methodology, and became systematic around the first world war. Without even knowing the content of encrypted messages, traffic analysts could map out an enemy “order of battle” or disposition of forces, and make inferences about commanders’ intentions.

Traffic analysis techniques can also cut through the petabytes of redundant babble and chatter in the financial and political worlds. Even with state secrecy and the forests of non-disclosure agreements around “proprietary” investment or trading algorithms, crowds can be remarkably revealing in their open-source posts on social media.

Predata, a three-year-old New York and Washington-based predictive data analytics provider, has a Princeton-intensive crew of engineers and international affairs graduates working on early “signals” of market and political events. Predata trawls the open metadata for users of Twitter, Wikipedia, YouTube, Reddit and other social media, and analyses it to find indicators of future price moves or official actions.

I have been following their signals for a while and find them to be useful indicators. Predata started by creating political risk indicators, such as Iran-Saudi antagonism, Italian or Chilean labour unrest, or the relative enthusiasm for French political parties. Since the beginning of this year, they have been developing signals for financial and commodities markets.

The 1-9-90 rule
1 per cent of internet users initiate discussions or content, 9 per cent transmit content or participate occasionally and 90 per cent are consumers or ‘lurkers’

Using the example of the company’s BoJ signal. For this, Predata collects the metadata from 300 sources, such as Twitter users, contested Wikipedia edits or YouTube items created by Japanese monetary policy geeks. Of those, at any time perhaps 100 are important, and 8 to 10 turn out to be predictive....This is where you need some domain knowledge [domain expertise = industry expertise]. It turns out that Twitter is pretty important for monetary policy, along with the Japanese-language Wiki page for the Bank of Japan, or, say, a YouTube video of [BoJ governor] Haruhiko Kuroda’s cross-examination before a Diet parliamentary committee.

“Then you build a network of candidate discussions [JK: training beds] and look for the pattern those took before historical moves. The machine-learning algorithm goes back and picks the leads and lags between traffic and monetary policy events.” [Jk: Large data sets with known correct answers serve as a training bed and then new data serves as a test bed]

Typically, Predata’s algos seem to be able to signal changes in policy or big price moves [jk: inflection points] somewhere between 2 days and 2 weeks in advance. Unlike some academic Twitter scholars, Predata does not do systematic sentiment analysis of tweets or Wikipedia edits. “We only look for how many people there are in the conversation and comments, and how many people disagreed with each other. We call the latter the coefficient of contestation,” Mr Shinn says.

The lead time for Twitter, Wiki or other social media signals varies from one market to another. Foreign exchange markets typically move within days, bond yields within a few days to a week, and commodities prices within a week to two weeks. “If nothing happens within 30 days,” says Mr Lee, “then we say we are wrong.”
algorithms  alternative_data  Bank_of_Japan  commodities  economics  economic_data  financial_markets  industry_expertise  inflection_points  intelligence_analysts  lead_time  machine_learning  massive_data_sets  metadata  non-traditional  Predata  predictive_analytics  political_risk  signals  social_media  spycraft  traffic_analysis  training_beds  Twitter  unconventional 
november 2017 by jerryking
Uber Extends an Olive Branch to Local Governments: Its Data
JAN. 8, 2017 | - The New York Times | By MIKE ISAAC.

unveiled Movement, a stand-alone website it hopes will persuade city planners to consider Uber as part of urban development and transit systems in the future.

The site, which Uber will invite planning agencies and researchers to visit in the coming weeks, will allow outsiders to study traffic patterns and speeds across cities using data collected by tens of thousands of Uber vehicles. Users can use Movement to compare average trip times across certain points in cities and see what effect something like a baseball game might have on traffic patterns. Eventually, the company plans to make Movement available to the general public.
municipalities  urban  urban_planning  cities  Boston  partnerships  Uber  Movement  data  data_driven  traffic_analysis 
january 2017 by jerryking
Sandy Pentland on the Social Data That Business Should Use - WSJ
Feb. 10, 2014 | Journal Report - CIO Netowrk| WSJ's Steve Rosenbush speaking with MIT's Sandy Pentland.

MR. ROSENBUSH: For most of us, social data is Twitter, it's Facebook. What do you mean by it?

MR. PENTLAND: Those sorts of things are people's public face. On the other hand, for instance, there's badge data. Every corporation has name badges. Many of these record where people come and go, door swipes and things like that. That's a different type of social media. Or if I look at cellphone data, I can tell when people get together, what they search for, who they talk to. You can look at connections between people in ways you never could before. The way most people approach this is incorrect, because they're asking questions about individuals. A better way to approach is asking questions about interactions between people.
social_data  interpretation  Twitter  Facebook  social_physics  Communicating_&_Connecting  informed_consent  location_based_services  data  massive_data_sets  contextual  LBMA  interactivity  traffic_analysis  mobile_phones 
february 2015 by jerryking
How should we analyse our lives? -
January 17, 2014 | FT |Gillian Tett.

“Social physics helps us understand how ideas flow from person to person . . . and ends up shaping the norms, productivity and creative output of our companies, cities and societies,” writes Pentland. “Just as the goal of traditional physics is to understand how the flow of energy translates into change in motion, social physics seems to understand how the flow of ideas and information translates into changes in behaviour.”...The only question now is whether these powerful new tools will be mostly used for good (to predict traffic queues or flu epidemics) or for more malevolent ends (to enable companies to flog needless goods, say, or for government control). Sadly, “social physics” and data crunching don’t offer any prediction on this issue, even though it is one of the dominant questions of our are always organised, collected and interpreted by people. Thus if you want to analyse what our interactions mean – let alone make decisions based on this – you will invariably be grappling with cultural and power relations.
massive_data_sets  social_physics  data_scientists  call_centres  books  data  social_data  traffic_analysis  flu_outbreaks  Gillian_Tett  queuing  quantified_self 
january 2014 by jerryking
Cellphone Data Track Our Migration Patterns -

Researchers at Massachusetts Institute of Technology are using international data flows of cellphone talk and Internet traffic to capture the complex social dynamics of urban life and globalization. Most recently, the researchers used anonymous real-time data supplied by AT&T Inc. on phone calls, web-browsing and email traffic to and from millions of New Yorkers to chart the city's global social networks. The resulting maps, called the New York Talk Exchange, were recently exhibited at the Museum of Modern Art.
data  migrants  MIT  mobile_phones  MoMA  New_York_City  patterns  real-time  social_networks  social_physics  tracking  traffic_analysis 
october 2011 by jerryking

Copy this bookmark:

to read