recentpopularlog in

jerryking : haystacks   17

Spy tactics can spot consumer trends
MARCH 22, 2016 | Financial Times | John Reed.
Israel’s military spies are skilled at sifting through large amounts of information — emails, phone calls, location data — to find the proverbial needle in a haystack: a suspicious event or anomalous pattern that could be the warning of a security threat.....So it is no surprise that many companies ask Israeli start-ups for help in data analysis. The start-ups, often founded by former military intelligence officers, are using the methods of crunching data deployed in spycraft to help commercial clients. These might range from businesses tracking customer behaviour to financial institutions trying to root out online fraud......Mamram is the Israel Defense Forces’ elite computing unit.
analytics  consumer_behavior  cyber_security  data  e-mail  haystacks  hedge_funds  IDF  insights  intelligence_analysts  Israel  Israeli  Mamram  maritime  massive_data_sets  security_&_intelligence  shipping  spycraft  start_ups  tracking  traffic_analysis  trends  trend_spotting 
april 2019 by jerryking
Little Brother
Sep 11th 2014 | The Economist | Alexandra Suich.

In 1963 David Ogilvy, the father of Madison Avenue and author of a classic business book, “Confessions of an Advertising Man”, wrote: “An advertisement is like a radar sweep, constantly hunting new prospects as they come into the market. Get a good radar, and keep it sweeping.”.....Behavioural profiling has gone viral across the internet, enabling firms to reach users with specific messages based on their location, interests, browsing history and demographic group......Extreme personalisation in advertising has been slow to come... online advertising space is unlimited and prices are low, so making money is not as easy as it was in the offline world,.....Digital advertising is being buoyed by three important trends. The first is the rise of mobile devices, such as smartphones....The second, related trend is the rise of social networks such as Facebook, Twitter and Pinterest, which have become an important navigation system for people looking for content across the web. ......The third big development has been the rise of real-time bidding, or “programmatic buying”, a new system for targeting consumers precisely and swiftly with online adverts. Publishers, advertisers and intermediaries can now bid for digital ads electronically and direct them to specific consumers at lightning speed.....The lines between established media businesses are becoming blurred. Richard Edelman, the boss of Edelman, a public-relations firm, describes the media and advertising business as a “mosh pit”. .... clients’ biggest question is whether people will even notice their ads. ...This special report will show that technology is profoundly changing the dynamics of advertising. Building on the vast amount of data produced by consumers’ digital lives, it is giving more power to media companies that have a direct relationship with their customers and can track them across different devices. ....Consumers may gain from advertising tailored to their particular needs, and so far most of them seem content to accept the ensuing loss of privacy. But companies are sensitive to the potential costs of overstepping the mark. As the head of one British advertising firm puts it: “Once people realise what’s happening, I can’t imagine there won’t be pushback.”
Facebook  Twitter  Pinterest  Ogilvy_&_Mather  David_Ogilvy  behavioural_targeting  pushback  books  effectiveness  haystacks  privacy  native_advertising  ad-tech  Conversant  Kraft  personalization  trends  mobile_phones  smartphones  social_media  real-time  auctions  programmatic  advertising  online_advertising  Omnicom 
february 2017 by jerryking
It’s a Whole New Data Game for Business - WSJ
Feb. 9, 2015 | WSJ |

opportunistic data collection is leading to entirely new kinds of data that aren’t well suited to the existing statistical and data-mining methodologies. So point number one is that you need to think hard about the questions that you have and about the way that the data were collected and build new statistical tools to answer those questions. Don’t expect the existing software package that you have is going to give you the tools you need....Point number two is having to deal with distributed data....What do you do when the data that you want to analyze are actually in different places?

There’s lots of clever solutions for doing that. But at some point, the volume of data’s going to outstrip the ability to do that. You’re forced to think about how you might, for example, reduce those data sets, so that they’re easier to move.
data  data_collection  datasets  data_mining  massive_data_sets  distributed_data  haystacks  questions  tools  unstructured_data 
february 2015 by jerryking
The value is in the details
November 30, 2012 | FT.com | By Ravi Mattu.

Troy Carter is the Founder and CEO, Atom Factory. He's also
Lady Gaga's manager used the web to help build her career and is turning his sights to big data.

One of those friends was Joe Lonsdale, co-founder of the Palo Alto-based data management company Palantir. “He said, ‘Send me all the data you have.’ So, we sent him everything and he said it was the worst data he had ever seen in his life.” The problem wasn’t the amount of data – they had lots of it, from Ticketmaster, Lady­gaga.com and merchandise sales – but the quality. Existing social media platforms weren’t much better. “When you deal with Facebook, the information you get is geographical – what city people are logging in from, what time of day – but you don’t get the behavioural information to help you build a better experience.”
massive_data_sets  music_industry  Lady_Gaga  data_driven  Facebook  African-Americans  behavioural_data  entrepreneur  data_quality  haystacks  data_management  customer_experience  detail_oriented  Palantir 
february 2013 by jerryking
What Data Can’t Do - NYTimes.com
By DAVID BROOKS
Published: February 18, 2013

there are many things big data does poorly. Let’s note a few in rapid-fire fashion:

* Data struggles with the social. Your brain is pretty bad at math (quick, what’s the square root of 437), but it’s excellent at social cognition. People are really good at mirroring each other’s emotional states, at detecting uncooperative behavior and at assigning value to things through emotion.
* Data struggles with context. Human decisions are embedded in contexts. The human brain has evolved to account for this reality...Data analysis is pretty bad at narrative and emergent thinking.
* Data creates bigger haystacks. This is a point Nassim Taleb, the author of “Antifragile,” has made. As we acquire more data, we have the ability to find many, many more statistically significant correlations. Most of these correlations are spurious and deceive us when we’re trying to understand a situation.
* Big data has trouble with big (e.g. societal) problems.
* Data favors memes over masterpieces. Data analysis can detect when large numbers of people take an instant liking to some cultural product. But many important (and profitable) products are hated initially because they are unfamiliar. [The unfamiliar has to accomplish behavioural change / bridge cultural divides]
* Data obscures hidden/implicit value judgements. I recently saw an academic book with the excellent title, “ ‘Raw Data’ Is an Oxymoron.” One of the points was that data is never raw; it’s always structured according to somebody’s predispositions and values. The end result looks disinterested, but, in reality, there are value choices all the way through, from construction to interpretation.

This is not to argue that big data isn’t a great tool. It’s just that, like any tool, it’s good at some things and not at others. As the Yale professor Edward Tufte has said, “The world is much more interesting than any one discipline.”
massive_data_sets  David_Brooks  data_driven  decision_making  data  Nassim_Taleb  contrarians  skepticism  new_graduates  contextual  risks  social_cognition  self-deception  correlations  value_judgements  haystacks  narratives  memes  unfamiliarity  naivete  hidden  Edward_Tufte  emotions  antifragility  behavioral_change  new_products  cultural_products  masterpieces  EQ  emotional_intelligence 
february 2013 by jerryking
Jake Porway, Data Scientist Information, Facts, News, Photos -- National Geographic
Data scientist Jake Porway (Ph.D.) is a matchmaker. He sees social change organizations working to make the world a better place, collecting mountains of data, but lacking skills and resources to use that information to advance their mission. He sees data scientists with amazing skills and cutting-edge tools, eager to use their talent to accomplish something meaningful, yet cut off from channels that allow them to do so. He sees governments ready to make data open and available, but disconnected from people who need it. For Porway, it's a match waiting to happen and the reason he founded DataKind (formerly Data Without Borders). It connects nonprofits, NGOs and other data-rich social change organizations with data scientists willing to donate time and knowledge to solve social, environmental and community problems. Ultimately, he wants to build a globally connected network of dedicated experts who can be deployed at a moment's notice to tackle any big data science task worldwide
data_scientists  DataKind  data  match-making  haystacks  PhDs  open_data  nonprofit  NGOs  volunteering 
july 2012 by jerryking
The Trouble with Big Data
May 5, 2012 | | What's The Big Data?| GilPress

“With too little data, you won’t be able to make any conclusions that you trust. With loads of data you will find relationships that aren’t real… On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough. You need experience in solving real world problems, because there are a lot of important limitations to the statistics that you learned in school. Big data isn’t about bits, it’s about talent.”.....The “talent” of “understanding the problem and the data applicable to it” is what makes a good scientist: The required skepticism, the development of hypotheses (models), and the un-ending quest to refute them, following the scientific method that has brought us remarkable progress over the course of the last three hundred and fifty years.
in_the_real_world  massive_data_sets  blogs  skepticism  challenges  problems  problem_solving  expertise  statistics  talent  spurious  data_quality  data_scientists  haystacks  correlations  limitations 
june 2012 by jerryking
Obsessed to a Fault - WSJ.com
APRIL 18, 2006 | WSJ | By LIAM PLEVEN. GeoVera, among the
largest sellers of quake insurance to homeowners on the open market in
California, is trying to buck the trend and make a profit at the same
time. About 40% of its business -- worth roughly $100 M in annual
premiums -- comes from selling quake insurance there. "What sets us
apart is our focus on catastrophe underwriting. That's all we do,"...The
company is a case study in the broader economics of disaster
insurance...GeoVera executives believe they can use their brains -- by
devouring data on the homes it insures, and keeping a close eye on the
location of its customers and the type of coverage they're buying -- to
make the company profitable. They analyze mounds of information about
the 115,000 homes the company insures: What are they made of? When were
they built? What types of foundations do they stand on? How solid is the
soil beneath them? Are they on a slope? How close are they to
California's 200-odd active faults?
catastrophes  insurance  disasters  GeoVera  underwriting  data_driven  risk-management  competingonanalytics  massive_data_sets  haystacks 
october 2010 by jerryking
Conquering the Digital Haystack, Search Techniques
Jan 1, 2005 | Inc. | Bob Buderi. "Will Google or Microsoft
buy Blinkx and crush it, or ignore it? Who knows? At the same time, all
the big players -- joined by firms like directory giant Dex Media and
online business service provider Interland -- have launched search
initiatives of their own. Many of these are aimed at small businesses,
including affordable fixed-price plans that guarantee certain numbers of
keyword clicks and local advertising programs. At the moment, it's the
trend that's important rather than any one particular offering. "
online_advertising  City_Voice  small_business  local_advertising  haystacks 
july 2010 by jerryking
For Today’s Graduate, Just One Word - Statistics - NYTimes.com
Aug. 5, 2009 | NYT | By STEVE LOHR. “We’re entering a world
where everything can be monitored and measured,” said Erik Brynjolfsson,
an economist and director of MIT’s Center for Digital Business. “But
the big problem is the ability of man to use, analyze and make sense of
the data.”" The rich lode of Web data has its perils. Its sheer vol. can
easily overwhelm statistical models. Statisticians caution that strong
correlations of data do not necessarily prove a cause-and-effect link.
E.g., in the late 1940s, before there was a polio vaccine, public health
experts noted that polio cases increased in step with the consumption
of ice cream and soft drinks, says David A. Grier, a historian and
statistician at GWU. Eliminating such treats was recommended as part of
an anti-polio diet. It turned out that polio outbreaks were most common
in the hot mths of summer, when people ate more ice cream, showing only
an association. The data explosion magnifies longstanding issues in
statistics.
Steve_Lohr  Hal_Varian  statistics  career_paths  haystacks  analytics  Google  data  Freshbooks  information_overload  data_scientists  Erik_Brynjolfsson  measurements  sense-making  massive_data_sets  correlations  causality 
june 2010 by jerryking
Needle in a haystack
Feb 27, 2010 | The Economist. Vol. 394, Iss. 8671; pg.
15.|Anonymous. AS DATA become more abundant, the main problem is no
longer finding the information as such but laying one's hands on the
relevant bits easily and quickly. What is needed is information about
information. Librarians and computer scientists call it metadata.
ProQuest  folksonomy  tagging  metadata  haystacks  commoditization_of_information  relevance 
march 2010 by jerryking
At a Software Powerhouse, the Good Life Is Under Siege
November 21, 2009 | New York Times |By STEVE LOHR. SAS’s
specialty, a lucrative niche called business intelligence software, is
becoming mainstream. Free, open-source alternatives to some of the
company’s products are increasingly popular. On the other end of the
spectrum, the heavyweights of the software industry — Oracle, SAP,
Microsoft and, especially, I.B.M. — are plunging in and investing
billions of dollars. As the stream of companies’ collected data turns
into a torrent, SAS and other software companies are trying to find new
ways to harness it.
Freshbooks  Steve_Lohr  competingonanalytics  data_driven  data_mining  SAS  haystacks 
november 2009 by jerryking
FT.com / Companies / Technology - Make sense of the in-house data mountain
November 22, 2006 | Financial Times | By Tom Braithwaite. With
swaths of unstructured data lying in corporate servers, whether in the
form of e-mails, PowerPoint presentations or TV images, companies are
increasingly seeking the means to sift through the in-house information
mountain.
search  in-house  databases  information_overload  haystacks  massive_data_sets  data_mining  unstructured_data  sense-making 
june 2009 by jerryking
I.B.M. Unveils Software to Find Trends in Vast Data Sets - NYTimes.com
May 20, 2009 | New York Times | By ASHLEE VANCE. New software
from I.B.M. can suck up huge volumes of data from many sources and
quickly identify correlations within it. The company says it expects the
software to be useful in analyzing finance, health care and even space
weather.
data_mining  IBM  pattern_recognition  massive_data_sets  Ashlee_Vance  haystacks 
may 2009 by jerryking

Copy this bookmark:





to read