Standards & Metrics – for the Localization Industry

Capita TI Strategic Account Director, Kris Smith, assesses the landscape of the current debate around measuring quality within the Localization industry.

There have been calls for the localization industry to work together to produce an agreed set of standards and metrics. ‘Quality’ is something which every Language Service Provider (LSP) offers to clients, but how do we define and measure this? And is the current approach most companies adopt still relevant in today’s Big Data landscape?

In the 1990s, widely used QA models such as SAE J2450 and the LISA QA were created. These models have not been updated and are now presented as “one-size-fits-all” models that do not reflect the needs of a rapidly diversifying translation industry. Speaking about these models, Jaap van der Meer from TAUS says, “This does not match today’s needs. We need to go up and down in quality, depending on the type of content and usage.”

There are ongoing efforts to create QA Tools which take into account the diversification of content. In June 2012, TAUS launched the Dynamic Quality Framework (DQF). TAUS DQF enables benchmarking, providing a framework of the best fit translation quality evaluation models based on content types, intended usage, tools, processes and other variables. It is a knowledge base documenting industry best practices for applying evaluation models and shared tools.

The Globalization and Localisation Association (GALA) have created QTLaunchPad – a European Commission funded collaborative research initiative dedicated to overcoming quality barriers in machine and human translation and in language technologies. It is preparing for a large-scale translation quality initiative for Europe and the QTLaunchPad consortium consists of several world-leading research centres.

Further research and efforts continue. Last week, the Centre for Next Generation Localization (CNGL) produced a white paper titled ‘Eye Tracking as an Automatic MT Evaluation Technique.” This is a potentially exciting drive to bring together eye tracking technology and Machine Translation (MT) to accurately assess raw MT output quality.

One of the challenges for the translation industry is that there isn’t one centralised governing body. TAUS, GALA. CNGL and many others are making noble efforts to create models and tools but each is following their own agenda. With multiple models we move further away from finding a centralized, industry standard metric.

This problem is one which is unlikely to be resolved in the short to medium term. Until MT starts to become embedded everywhere then I believe LSP’s will continue to produce their own qualimetrics, bespoke for each client. This can be seen in the recent study by Stephen Doherty at the CNGL which shows that there is a clear preference for using internal models amongst LSP’s. Companies may also elect to use tools such as TAUS DQF, QTLaunchPad and combine these tools with emerging technologies such as eye tracking in order to enhance their quality offering and internal QA processes.

Will we reach a point where we have an agreed set of industry wide standards and metrics, flexible enough to cope with different types of content and usage? This will require collaboration on a scale never before seen in our industry and will require leadership and a compelling reason for change. Stephen Doherty’s report shows that the majority (70%) of the 313 respondents reported a need for gradual improvements, e.g. by developing better quality evaluation software or adopting standard metrics, so the recognition is there. This is an important first step on what could be the road to standardization.

kris.smith@capita-ti.com

@L10nKris

 

Resources

Jaap van der Meer, Director at TAUS (http://www.translationautomation.com/press-releases/taus-launchesdynamic-quality-evaluation-framework)

Stephen Doherty, Sharon O’Brien, Michael Carl, (http://openarchive.cbs.dk/bitstream/handle/10398/8045/SubmissionforMT_dohertyobriencarl.pdf?sequence=1)

Stephen Doherty, Translation Quality Models and Tools – Is There Room for Improvement? (http://www.gala-global.org/blog/2013/translation-quality-models-and-tools-is-there-room-for-improvement/)

 

 


Share this story

Sign up to our newsletter

Sign up