tag:blogger.com,1999:blog-33553336932466148462024-03-28T02:36:25.079-07:00This is StuffMerihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.comBlogger38125tag:blogger.com,1999:blog-3355333693246614846.post-58254801666773433982020-04-18T03:44:00.001-07:002020-05-28T00:45:36.175-07:00Open Link in New Tab<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><br />
<div style="text-align: justify;">All major browsers use popup blockers to prevent sites from opening new tabs. Popup blockers works in slightly different ways and block popups in different contexts. <br />
<br />
First chapter gives short overview of popup blockers. The rest of article shows multiple ways how to open new tab and how to work around browsers limitations.<br />
<a name='more'></a><br />
<a name="TableOfContents"></a><h4>Table Of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><ul><li><a href="#TableOfContents">Table Of Contents</a></li>
<li><a href="#PopupBlockers">Popup Blockers</a></li>
<li><a href="#HtmlLink">Html Link</a></li>
<li><a href="#JavaScript">JavaScript</a></li>
<ul><li><a href="#JavaScriptWhentheUrlisKnown">When the Url is Known</a></li>
<ul><li><a href="#JavaScriptWhentheUrlisKnownFromClick">From Click</a></li>
<li><a href="#JavaScriptWhentheUrlisKnownFromTimerInsideClick">From Timer Inside Click</a></li>
<li><a href="#JavaScriptWhentheUrlisKnownFromMouseOver">From Mouse Over</a></li>
<li><a href="#JavaScriptWhentheUrlisKnownFromTimerInsideMouseOver">From Timer Inside Mouse Over</a></li>
<li><a href="#JavaScriptWhentheUrlisKnownFromBodyOnload">From Body Onload</a></li>
</ul><li><a href="#JavaScriptFetchingUrlFromBackend">Fetching Url From Backend</a></li>
<ul><li><a href="#JavaScriptFetchingUrlFromBackendFromRequestResponse">From Request Response</a></li>
<li><a href="#JavaScriptFetchingUrlFromBackendWorkaround1">Workaround 1</a></li>
<li><a href="#JavaScriptFetchingUrlFromBackendWorkaround2">Workaround 2</a></li>
</ul><li><a href="#JavaScriptFewNotesonOpenFunction">Few Notes on Open Function</a></li>
</ul><li><a href="#AllExamples">All Examples</a></li>
</ul></div></div><a name="PopupBlockers"></a><h4>Popup Blockers</h4>A browser does not distinguish between a new tab and a popup. Therefore popup blockers treat them the same way.<br />
and in this article I use the terms "new tab" and "new window" interchangeably.<br />
<br />
Popup blocker built inside the Firefox is willing to open new tab only from onclick hander. It prevents tab opening from mouse over event, network request response, and from inside script in the body. Instead of opening new tab, firefox will show yellow warning banner which says <code>Firefox prevented this site from opening popup</code>. It is possible to turn off the warning, but that requires user action. <br />
<br />
Edge works similarly to Firefox, except that banner is white and show on bottom of the window.<br />
<br />
Chrome always prevents new tabs from network request response. It mostly prevents also new tabs from on mouse over event and script inside the body. The last two are sometimes created, but I did not found when exactly. Chrome shows small icon in url bar when it refuses to open the new tab. User can click on it and allow popups manually.<br />
<br />
<a name="HtmlLink"></a><h4>Html Link</h4>The easiest way to create a link that opens in a new tab is to add <code>target="_blank"</code> to it. It works in Chrome, Firefox and Edge. <br />
<div style="text-align: left;"><pre class="brush:text"><a href="http://gooogle.com" target="_blank">Normal Link</a></pre></div>This is how it renders: <a href="http://gooogle.com" target="_blank">Normal Link</a><br />
<br />
<a name="JavaScript"></a><h4>JavaScript</h4>JavaScript can open new tab using the <code>window.open(URL, name, specs, replace)</code> function. The trick is to put <code>'_blank'</code> into <code>name</code> argument. <br />
<div style="text-align: left;"><pre class="brush:xml">window.open('https://www.google.com', '_blank');
</pre></div>This works in most, but not all, situations. Most importantly, Edge and Firefox block new tab from network request response handler. When the url have to be loaded from backend, Firefox requires tricks to work.<br />
<br />
This chapter has two sections. <a href="#JavaScriptWhentheUrlisKnown">First section</a> deals with situation when the frontend knows the url. <a href="#JavaScriptFetchingUrlFromBackend">Second section</a> shows how to open link in new tab when the fronted needs to send network request.<br />
<br />
<a name="JavaScriptWhentheUrlisKnown"></a><h5>When the Url is Known</h5>In this chapter, we will try to open a new tab from mouse click, from timer, from mouse over and from a script inside the body. <br />
<br />
<a name="JavaScriptWhentheUrlisKnownFromClick"></a><h6>From Click</h6>Open new tab from <code>onclick</code> attribute of html tag works from Chrome, Firefox and Edge:<br />
<div style="text-align: left;"><pre class="brush:xml"><a href="#" onclick="
window.open('https://www.google.com', '_blank');
return false;
">on click</a></pre></div>This is how it renders: <a href="#" onclick="window.open('https://www.google.com', '_blank'); return false;">on click</a><br />
<br />
<a name="JavaScriptWhentheUrlisKnownFromTimerInsideClick"></a><h6>From Timer Inside Click</h6>You can also start a timer while handling users onclick method and open a new tab later from the timer. It works in Chrome, Firefox and Edge:<br />
<div style="text-align: left;"><pre class="brush:xml"><a href="#" onclick="
setTimeout(() => window.open('https://www.google.com', '_blank'), 500);
return false;
">click starts timer</a>
</pre></div>This is how it renders: <a href="" onclick="setTimeout(function() {window.open('https://www.google.com', '_blank');}, 500); return false;">click starts timer</a><br />
<br />
<a name="JavaScriptWhentheUrlisKnownFromMouseOver"></a><h6>From Mouse Over</h6>Open new tab from <code>onmouseover</code> attribute of an html tag does not work. Firefox and Edge block it all the time. Chrome blocks it most of the time, although blocking is not perfect and the tab occasionally shows up: <div style="text-align: left;"><pre class="brush:xml"><a href="#" onmouseover="
window.open('https://www.google.com', '_blank');
return false;
">on mouse over</a></pre></div>This is how it renders: <a href="" onmouseover="window.open('https://www.google.com', '_blank'); return false;">on mouse over</a><br />
<br />
<a name="JavaScriptWhentheUrlisKnownFromTimerInsideMouseOver"></a><h6>From Timer Inside Mouse Over</h6>Open new tab from timer started inside <code>onmouseover</code> does not work. Firefox and Edge block it all the time. Chrome blocks it most of the time, although blocking is not perfect and the tab shows up once in a while:<div style="text-align: left;"><pre class="brush:xml"><a href="#" onmouseover="
setTimeout(function() {
window.open('https://www.google.com', '_blank');
} ,500);
return false;
">timer on mouseover</a></br>
</pre></div>This is how it renders: <a href="#" onmouseover="
setTimeout(function() {
window.open('https://www.google.com', '_blank');
} ,500);
return false;
">timer on mouseover</a><br />
<br />
<a name="JavaScriptWhentheUrlisKnownFromBodyOnload"></a><h6>From Body Onload</h6>Opening new tab from script in body is blocked by all three browser. Add this inside the body tag to see it: <div style="text-align: left;"><pre class="brush:xml"><script>
console.log("installing");
setTimeout(function() {
console.log("running");
window.open('https://www.google.com', '_blank');
}, 500);
</script></pre></div><br />
<a name="JavaScriptFetchingUrlFromBackend"></a><h5>Fetching Url From Backend</h5>More interesting and useful case is when the web application needs to fetch the url from backend. The straightforward solution does not work, Edge and Firefox block the new tab. However, this use case is too useful to be ignored, so we will build workaround. <br />
<br />
<a name="JavaScriptFetchingUrlFromBackendFromRequestResponse"></a><h6>From Request Response</h6>Opening new tab from request response works only in Chrome. Edge and Firefox shows yellow warning and prevents new tab to open:<div style="text-align: left;"><pre class="brush:js"><script>
function fromHttpRequest() {
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange = (e) => {
console.log(Http.responseText)
window.open('https://www.google.com', '_blank');
}
}
</script>
<a href="#" onclick="
fromHttpRequest();
return false;
">from http request</a></br></pre></div>This is how it renders: <script>
function fromHttpRequest() {
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange = (e) => {
console.log(Http.responseText)
window.open('https://www.google.com', '_blank');
}
}
</script><a href="" onclick="fromHttpRequest(); return false">from http request</a><br />
<br />
Starting timer from request response does not help. Chrome and Edge will open the tab, but Firefox will show warning instead. I have omitted the example, because it is long and almost the same as timers above.<br />
<br />
<a name="JavaScriptFetchingUrlFromBackendWorkaround1"></a><h6>Workaround 1</h6>The workaround is to combine javascript interval started from user click, network request and let them communicate through variable.<br />
<ul><li>Network request waits for response. Then it sets the common variable.</li>
<li>The interval periodically checks variable value. It opens new tab once the variable has the response. Then the interval removes itself.</li>
</ul><div style="text-align: left;"><pre class="brush:xml"><script>
function intervalListensForRequestResponse() {
var iHaveTheResponse; // shared variable
// send request
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange = (e) => {
console.log(Http.responseText)
iHaveTheResponse = 'https://www.google.com';
}
// periodically check the variable
var refreshIntervalId = setInterval(function(){
console.log('tick ' + iHaveTheResponse);
if (iHaveTheResponse) {
window.open(iHaveTheResponse, '_blank')
clearInterval(refreshIntervalId);
}
}, 500);
return false;
}
</script>
<a href="#" onclick="
return intervalListensForRequestResponse();
">interval listens for request response</a>
</pre></div>This is how it renders: <script>
function intervalListensForRequestResponse() {
var iHaveTheResponse; // shared variable
// send request
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange = (e) => {
console.log(Http.responseText)
iHaveTheResponse = 'https://www.google.com';
}
// periodically check the variable
var refreshIntervalId = setInterval(function(){
console.log('tick ' + iHaveTheResponse);
if (iHaveTheResponse) {
// open new tab
window.open(iHaveTheResponse, '_blank')
// remove the interval
clearInterval(refreshIntervalId);
}
}, 500);
return false;
}
</script> <a href="#" onclick="return intervalListensForRequestResponse();">interval listens for request response</a><br />
<br />
<a name="JavaScriptFetchingUrlFromBackendWorkaround2"></a><h6>Workaround 2</h6>If the frontend knows for sure it will have to open new tab and just needs to acquire correct url from backend, you can:<br />
<ol><li>Open new window and keep reference to it.</li>
<li>Send the request.</li>
<li>Set url to newly opened window.</li>
</ol><br />
The disadvantage of this solution is that new window is opened before the response from backend came back. There is a blank window for few miliseconds. In addition, it is impossible to make backend decide whether the window will be needed or not. <div style="text-align: left;"><pre class="brush:js"><script>
function startByOpeningTheWindow() {
var newTab = window.open("", '_blank');
if (newTab==null) {
return ;
}
// send request
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange = (e) => {
console.log(Http.responseText);
newTab.location.href = 'https://www.google.com';
}
return false;
}
</script>
<a href="#"
onclick="startByOpeningTheWindow();"
>first open the window, change url later</a></br></pre></div>This is how it renders: <script>
function startByOpeningTheWindow() {
var newTab = window.open("", '_blank');
if (newTab==null) {
return ;
}
// send request
const Http = new XMLHttpRequest();
const url='https://jsonplaceholder.typicode.com/posts';
Http.open("GET", url);
Http.send();
Http.onreadystatechange = (e) => {
console.log(Http.responseText);
newTab.location.href = 'https://www.google.com';
}
return false;
}
</script><a href="#" onclick="startByOpeningTheWindow();">first open the window, change url later</a><br />
<br />
<a name="JavaScriptFewNotesonOpenFunction"></a><h5>Few Notes on Open Function</h5>Communication between original window and new tab is possible if they have the same origin. If they show pages from different domains, the communication is blocked by the browser due to Same Origin Policy rules. It ends with <code>Error: uncaught exception: Permission denied to get property <property_name or method_name></code> error.<br />
<br />
The <code>window.open</code> function returns reference to the tab that was just opened. The original page can use it to call functions inside newly opened tab. If the Firefox popup blocker blocked the popup, the function returns <code>null</code>. You can use this to check whether the new tab was blocked. This method works only for built-in popup blocker, it does not work for ad-blocking addons. <br />
<br />
New tab has the <code>opener</code> property which can be used to call functions inside the main window. It also has the <code>closed</code> property. Main window can use it to check whether the child tab was open or not.<br />
<br />
<a name="AllExamples"></a><h4>All Examples</h4>All tested cases are on <a href="https://sommeri.github.io/simple-samples-js/open-link-in-new-tab/open-link-in-new-tab.html">this page</a>.<br />
<br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com29tag:blogger.com,1999:blog-3355333693246614846.post-12507259712059954752015-12-12T07:42:00.001-08:002015-12-12T07:42:24.688-08:00Speedwriting<div style="text-align: justify;">Speedwriting is simple fast paced typing game with a twist - reading is not always easy. Read and write letters you see as quickly as possible anyway. The game can be played on <a href="http://www.newgrounds.com/portal/view/664163">newgrounds</a> or on <a href="http://sommeri.github.io/JSPrototype/web/minithings/3%20-%20Speedwriting/">Github</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTr5UGZKlfDdrfrOLVlW_1B51F8sH1UqD0qtpBex2l63AztBnCf1DUDgBVvVmerVEmyecG8ml2HtSXv2tqWqMUD21BIzeBvv15kUIDEPa_nvad-rYzYw5j7AqSh_lELddo1gU73RaTNeGv/s1600/turn_it_round.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTr5UGZKlfDdrfrOLVlW_1B51F8sH1UqD0qtpBex2l63AztBnCf1DUDgBVvVmerVEmyecG8ml2HtSXv2tqWqMUD21BIzeBvv15kUIDEPa_nvad-rYzYw5j7AqSh_lELddo1gU73RaTNeGv/s320/turn_it_round.png" /></a></div><br />
All my games can be found <a href="http://sommeri.github.io/JSPrototype/">Games & Stuff</a> page.</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com55tag:blogger.com,1999:blog-3355333693246614846.post-60859642803822589762015-01-26T05:12:00.000-08:002015-01-26T05:16:44.319-08:00Testing Grunt Plugin From Grunt<style>div.blogger-clickTrap {display: none;}</style><div style="text-align: justify;">Writing tests for grunt plugin turned out to be less straightforward then expected. I needed to run multiple task configurations and wanted to invoke them all by typing <code>grunt test</code> in main directory. <br />
<br />
Grunt normally exits after first task failure. That makes it impossible to store multiple failure scenarios inside the main project gruntfile. Running them from there would require the <code>--force</code> option, but grunt then ignores all warnings which is not optimal.<br />
<br />
Cleaner solution is to have a bunch of gruntfiles in separate directory and invoke them all from the main project gruntfile. This post explains how to do that.<a name='more'></a><br />
<br />
<a name="TableofContents"></a><h4>Table of Contents</h4><ul><li><a href="#DemoProject">Demo Project</a></li>
<li><a href="#RunningGruntfileFromSubdirectory">Running Gruntfile From Subdirectory</a></li>
<ul><li><a href="#RunningGruntfileFromSubdirectoryTheProblem">The Problem</a></li>
<li><a href="#RunningGruntfileFromSubdirectoryExplanation">Explanation</a></li>
<li><a href="#RunningGruntfileFromSubdirectorySolution1:DuplicateNpmRepository">Solution 1: Duplicate Npm Repository</a></li>
<li><a href="#RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectory">Solution 2: Load Grunt Tasks From Parent Directory</a></li>
</ul><li><a href="#CallingGruntfileFromJavascript">Calling Gruntfile From Javascript</a></li>
<ul><li><a href="#CallingGruntfileFromJavascriptExec">Exec</a></li>
<ul><li><a href="#CallingGruntfileFromJavascriptExecCallExec">Call Exec</a></li>
<li><a href="#CallingGruntfileFromJavascriptExecUnitTests">Unit Tests</a></li>
<li><a href="#CallingGruntfileFromJavascriptExecDisadvantages">Disadvantages</a></li>
</ul><li><a href="#CallingGruntfileFromJavascriptFork">Fork</a></li>
<ul><li><a href="#CallingGruntfileFromJavascriptForkCallGrunt">Call Grunt</a></li>
<li><a href="#CallingGruntfileFromJavascriptForkModuleArguments">Module Arguments</a></li>
<li><a href="#CallingGruntfileFromJavascriptForkCallFork">Call Fork</a></li>
<li><a href="#CallingGruntfileFromJavascriptForkWriteTests">Write Tests</a></li>
<li><a href="#CallingGruntfileFromJavascriptForkDisadvantages">Disadvantages</a></li>
</ul><li><a href="#CallingGruntfileFromJavascriptSpawn">Spawn</a></li>
<ul><li><a href="#CallingGruntfileFromJavascriptSpawnTheProblem">The Problem</a></li>
<li><a href="#CallingGruntfileFromJavascriptSpawnLoopingSuffixes">Looping Suffixes</a></li>
<li><a href="#CallingGruntfileFromJavascriptSpawnWriteTests">Write Tests</a></li>
<li><a href="#CallingGruntfileFromJavascriptSpawnDisadvantages">Disadvantages</a></li>
</ul></ul><li><a href="#Conclusion">Conclusion</a></li>
</ul><br />
<a name="DemoProject"></a><h4>Demo Project</h4><a href="https://github.com/SomMeri/grunt-plugin-testing-demo">Demo project</a> is small grunt plugin with one grunt task. The task either fails with warning or prints success message into the console depending on the value of <code>action</code> options property.<br />
<br />
The task:<br />
<div style="text-align: left;"><pre class="brush:js">grunt.registerMultiTask('plugin_tester', 'Demo grunt task.', function() {
//merge supplied options with default options
var options = this.options({ action: 'pass', message: 'unknown error'});
//pass or fail - depending on configured options
if (options.action==='pass') {
grunt.log.writeln('Plugin worked correctly passed.');
} else {
grunt.warn('Plugin failed: ' + options.message);
}
});
</pre></div>There are three different ways how to write grunt plugin unit tests. Each solution has its own nodeunit file in <code>test</code> directory and is explained in this post:<ul><li><a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/master/test/plugin_exec_test.js">plugin_exec_test.js</a> - the most practical <a href="#CallingGruntfileFromJavascriptExec">solution</a>,</li>
<li><a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/master/test/plugin_fork_test.js">plugin_fork_test.js</a> - <a href="#CallingGruntfileFromJavascriptFork">solves</a> rare edge case where previous solution fails,</li>
<li><a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/master/test/plugin_spawn_test.js">plugin_spawn_test.js</a> - <a href="#CallingGruntfileFromJavascriptSpawn">possible</a>, but least practical.</li>
</ul><br />
All three demo tests consist of three different task configurations:<pre class="brush:js">// Success scenario
options: { action: 'pass' }
// Fail with "complete failure" message
options: { action: 'fail', message: 'complete failure' }
//Fail with "partial failure" message
options: { action: 'fail', message: 'partial failure' }</pre><br />
Each configuration is stored in separate gruntfile inside <code>test</code> directory. For example, success scenario stored inside <code>gruntfile-pass.js</code> file looks like this:<br />
<div style="text-align: left;"><pre class="brush:js">grunt.initConfig({
// prove that npm plugin works too
jshint: {
all: [ 'gruntfile-pass.js' ]
},
// Configuration to be run (and then tested).
plugin_tester: {
pass: { options: { action: 'pass' } }
}
});
// Load this plugin's task(s).
grunt.loadTasks('./../tasks');
// next line does not work - grunt requires locally installed plugins
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.registerTask('default', ['plugin_tester', 'jshint']);
</pre></div>All three test gruntfiles look almost the same, only the <code>options</code> object of <code>plugin_tester</code> target changes. <br />
<br />
<a name="RunningGruntfileFromSubdirectory"></a><h4>Running Gruntfile From Subdirectory</h4>Our test gruntfiles are stored in <code>test</code> subdirectory and grunt does not handle such situation well. This chapter explains what the problem is and shows two ways how to solve it.<br />
<br />
<a name="RunningGruntfileFromSubdirectoryTheProblem"></a><h5>The Problem</h5>To see the problem, go to the demo project directory and run following command:<br />
<div style="text-align: left;"><pre class="brush:text">grunt --gruntfile test/gruntfile-problem.js</pre></div>Grunt responds with following error:<br />
<div style="text-align: left;"><pre class="brush:text">Local Npm module "grunt-contrib-jshint" not found. Is it installed?
Warning: Task "jshint" not found. Use --force to continue.
Aborted due to warnings.
</pre></div><a name="RunningGruntfileFromSubdirectoryExplanation"></a><h5>Explanation</h5>Grunt assumes that grunfile and node_modules repository are stored in the same directory. While node.js <code>require</code> function searches all parent directories for required module, grunts <code>loadNpmTasks</code> does not. <br />
<br />
This problem has two possible solutions, simple and fancy one:<ul><li>create local npm repository in tests directory (<a href="#RunningGruntfileFromSubdirectorySolution1:DuplicateNpmRepository">simple</a>),</li>
<li>make grunt load tasks from parent directories (<a href="#RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectory">fancy</a>).</li>
</ul><br />
Although the first "simple" solution is somewhat cleaner, demo project uses second "fancy" solution.<br />
<br />
<a name="RunningGruntfileFromSubdirectorySolution1:DuplicateNpmRepository"></a><h5>Solution 1: Duplicate Npm Repository</h5>The main idea is simple, just create another local npm repository inside the tests directory:<ul><li>Copy <code>package.json</code> file into <code>tests</code> directory.</li>
<li>Add test only dependencies into it.</li>
<li>Run <code>npm install</code> command every time you run tests.</li>
</ul><br />
This is the cleaner solution. It has only two downsides:<ul><li>test dependencies have to be maintained separately,</li>
<li>all plugin dependencies have to be installed in two places.</li>
</ul><br />
<a name="RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectory"></a><h5>Solution 2: Load Grunt Tasks From Parent Directory</h5>The other solution is to force grunt to load tasks from npm repository stored inside another directory.<br />
<br />
<a name="RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectoryGruntPluginLoading"></a><h6>Grunt Plugin Loading</h6>Grunt has two methods able to load plugins:<ul><li><code>loadTasks('directory-name')</code> - loads all tasks inside a directory,</li>
<li><code>loadNpmTasks('plugin-name')</code> - loads all tasks defined by a plugin.</li>
</ul><br />
The <a href="https://github.com/gruntjs/grunt/blob/09f8fc498b570ef1ed8f8e61a5ef63efef82fc76/lib/grunt/task.js#L382-L414"><code>loadNpmTasks</code></a> function assumes fixed directory structure of both grunt plugin and modules repository. It guess name of directory where tasks should be stored and then calls <code>loadTasks('directory-name')</code> function. <br />
<br />
Local npm repository has separate subdirectory for each npm package. All grunt plugins are supposed to have <code>tasks</code> subdirectory and <code>.js</code> files inside it are assumed to contain tasks. For example, <code>loadNpmTasks('grunt-contrib-jshint')</code> call loads tasks from <code>node_mudules/grunt-contrib-jshint/tasks</code> directory and is equivalent to:<br />
<div style="text-align: left;"><pre class="brush:js">grunt.loadTasks('node_modules/grunt-contrib-jshint/tasks')
</pre></div>Therefore, if we want to load all tasks of <code>grunt-contrib-jshint</code> plugin from parent directory, we can do following: <br />
<div style="text-align: left;"><pre class="brush:js">grunt.loadTasks('../node_modules/grunt-contrib-jshint/tasks')
</pre></div><a name="RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectoryLoopParentDirectories"></a><h6>Loop Parent Directories</h6>More flexible solution is to climb through all parent directories until we find closest node_modules repository or reach root directory. This is implemented inside <a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/19176bb0fe680fc0520587791d6f9e89cb0e5183/test/grunt-hacks.js"><code>grunt-hacks.js</code></a> module. <br />
<br />
The <code>loadParentNpmTasks</code> function loops parent directories :<br />
<div style="text-align: left;"><pre class="brush:js">module.exports = new function() {
this.loadParentNpmTasks = function(grunt, pluginName) {
var oldDirectory='', climb='', directory, content;
// search for the right directory
directory = climb+'node_modules/'+ pluginName;
while (continueClimbing(grunt, oldDirectory, directory)) {
climb += '../';
oldDirectory = directory;
directory = climb+'node_modules/'+ pluginName;
}
// load tasks or return an error
if (grunt.file.exists(directory)) {
grunt.loadTasks(directory+'/tasks');
} else {
grunt.fail.warn('Tasks plugin ' + pluginName + ' was not found.');
}
}
function continueClimbing(grunt, oldDirectory, directory) {
return !grunt.file.exists(directory) &&
!grunt.file.arePathsEquivalent(oldDirectory, directory);
}
}();
</pre></div><a name="RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectoryModifiedGruntfile"></a><h6>Modified Gruntfile</h6>Finally, we need to replace the usual <code>grunt.loadNpmTasks('grunt-contrib-jshint')</code> call in the gruntfile by following:<br />
<div style="text-align: left;"><pre class="brush:js">var loader = require("./grunt-hacks.js");
loader.loadParentNpmTasks(grunt, 'grunt-contrib-jshint');
</pre></div><br />
Shortened gruntfile:<br />
<div style="text-align: left;"><pre class="brush:js">module.exports = function(grunt) {
var loader = require("./grunt-hacks.js");
grunt.initConfig({
jshint: { /* ... */ },
plugin_tester: { /* ... */ }
});
grunt.loadTasks('./../tasks');
loader.loadParentNpmTasks(grunt, 'grunt-contrib-jshint');
};
</pre></div><br />
<a name="RunningGruntfileFromSubdirectorySolution2:LoadGruntTasksFromParentDirectoryDisadvantages"></a><h6>Disadvantages</h6>This solution has two disadvantages:<ul><li>It does not deal with collection plugins.</li>
<li>If grunt ever changes expected structure of grunt plugins, you will have to modify the solution.</li>
</ul><br />
If you need collection plugins too, have a look at grunts <a href="https://github.com/gruntjs/grunt/blob/09f8fc498b570ef1ed8f8e61a5ef63efef82fc76/lib/grunt/task.js#L388-L405">task.js</a> to see how to support them.<br />
<br />
<a name="CallingGruntfileFromJavascript"></a><h4>Calling Gruntfile From Javascript</h4>Second thing we need to do is to invoke the gruntfile from javascript. The only complication is that grunt exits whole process on task failure. Therefore, we need to call it from child process. <br />
<br />
Node module <a href="http://nodejs.org/api/child_process.html">child process</a> has three different functions able to run command inside child process:<ul><li><code>exec</code> - executes command on command line,</li>
<li><code>spawn</code> - differently executes command on command line,</li>
<li><code>fork</code> - runs node module in child process.</li>
</ul><br />
The first one, <code>exec</code>, is easiest to use and is explained in the <a href="#CallingGruntfileFromJavascriptExec">first</a> subchapter. <a href="#CallingGruntfileFromJavascriptFork">Second</a> subchapter shows how to use fork and why it is less optimal then exec. <a href="#CallingGruntfileFromJavascriptSpawn">Third</a> subchapter is about spawn.<br />
<br />
<a name="CallingGruntfileFromJavascriptExec"></a><h5>Exec</h5><a href="http://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback">Exec</a> runs command line command inside a child process. You can specify in which directory to run it, set up environment variables, set timeout after which the command will be killed and so on. When the command finishes its run, exec calls callback and passes it stdout stream, stderr streams and error if the command crashed.<br />
<br />
Unless configured otherwise, command is run in current directory. We want it to run inside <code>tests</code> subdirectory, so we have to specify <code>cwd</code> property of options object: <code>{cwd: 'tests/'}</code>.<br />
<br />
Both stdout and stderr streams content are stored inside a buffer. Each buffer has maximum size set to 204800 and if the command produces more output, <code>exec</code> call will crash. That amount is enough for our small task. If you need more you have to set <code>maxBuffer</code> options property.<br />
<br />
<a name="CallingGruntfileFromJavascriptExecCallExec"></a><h6>Call Exec</h6>Following code snippet shows how to run the gruntfile from exec. The function is asynchronous and calls <code>whenDoneCallback</code> after all is done:<br />
<div style="text-align: left;"><pre class="brush:js">var cp = require("child_process");
function callGruntfile(filename, whenDoneCallback) {
var command, options;
command = "grunt --gruntfile "+filename+" --no-color";
options = {cwd: 'test/'};
cp.exec(command, options, whenDoneCallback);
}
</pre></div>Note: if you installed npm into tests directory (<a href="#RunningGruntfileFromSubdirectorySolution1:DuplicateNpmRepository">simple solution</a>), then you need to use <code>callNpmInstallAndGruntfile</code> function instead of <code>callGruntfile</code>:<br />
<div style="text-align: left;"><pre class="brush:js">function callNpmInstallAndGruntfile(filename, whenDoneCallback) {
var command, options;
command = "npm install";
options = {cwd: 'test/'};
cp.exec(command, {}, function(error, stdout, stderr) {
callGruntfile(filename, whenDoneCallback);
});
}
</pre></div><a name="CallingGruntfileFromJavascriptExecUnitTests"></a><h6>Unit Tests</h6>First node unit test runs success scenario and then checks whether process finished without failure, whether standard output contains expected message and whether standard error is empty.<br />
<br />
Success scenario unit test:<br />
<div style="text-align: left;"><pre class="brush:js">pass: function(test) {
test.expect(3);
callGruntfile('gruntfile-pass.js', function (error, stdout, stderr) {
test.equal(error, null, "Command should not fail.");
test.equal(stderr, '', "Standard error stream should be empty.");
var stdoutOk = contains(stdout, 'Plugin worked correctly.');
test.ok(stdoutOk, "Missing stdout message.");
test.done();
});
},
</pre></div>Second node unit test runs "complete failure" scenario and then checks whether process failed as expected. Note that standard error stream is empty and warnings are printed into standard output.<br />
<br />
Failing scenario unit test:<br />
<div style="text-align: left;"><pre class="brush:js">fail_1: function(test) {
test.expect(3);
var gFile = 'gruntfile-fail-complete.js';
callGruntfile(gFile, function (error, stdout, stderr) {
test.equal(error, null, "Command should have failed.");
test.equal(error.message, 'Command failed: ', "Wrong error message.");
test.equal(stderr, '', "Non empty stderr.");
var stdoutOk = containsWarning(stdout, 'complete failure');
test.ok(stdoutOk, "Missing stdout message.");
test.done();
});
}
</pre></div>Third "partial failure" node unit test is almost the same as the previous one. Whole tests file is available <a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/19176bb0fe680fc0520587791d6f9e89cb0e5183/test/plugin_exec_test.js">on github</a>.<br />
<br />
<a name="CallingGruntfileFromJavascriptExecDisadvantages"></a><h6>Disadvantages</h6>Disadvantage:<ul><li>Maximum buffer size must be set in advance.</li>
</ul><br />
<a name="CallingGruntfileFromJavascriptFork"></a><h5>Fork</h5><a href="http://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options">Fork</a> runs node.js module inside child process and is equivalent to calling <code>node <module-name></code> on command line. Fork uses callbacks to send standard output and standard error to caller. Both callbacks can be called many times and caller obtains child process outputs in pieces.<br />
<br />
Using fork makes sense only if you need to handle arbitrary sized stdout and stderr or if you need to customize grunt functionality. If you do not, <code>exec</code> is easier to use.<br />
<br />
This chapter is split into four sub-chapters: <ul><li><a href="#CallingGruntfileFromJavascriptForkCallGrunt">call grunt</a> from javascript,</li>
<li><a href="#CallingGruntfileFromJavascriptForkModuleArguments">read command</a> line arguments inside node module,</li>
<li><a href="#CallingGruntfileFromJavascriptForkCallFork">start node</a> module inside a child process,</li>
<li><a href="#CallingGruntfileFromJavascriptForkWriteTests">write</a> unit tests.</li>
</ul><br />
<a name="CallingGruntfileFromJavascriptForkCallGrunt"></a><h6>Call Grunt</h6>Grunt was not meant to be called programatically. It does not expose "public" API and does not document it. <br />
<br />
Our solution mimics what grunt-cli does, so it is relatively future safe. Grunt-cli is distributed separately from grunt core and therefore is less likely to change. However, if it does change, this solution will have to change too. <br />
<br />
Running grunt from javascript requires us to:<ul><li>separate gruntfile name from its path,</li>
<li>change active directory,</li>
<li>call grunts <code>tasks</code> function.</li>
</ul><br />
Call grunt from javascript:<br />
<div style="text-align: left;"><pre class="brush:js">this.runGruntfile = function(filename) {
var grunt = require('grunt'), path = require('path'), directory, filename;
// split filename into directory and file
directory = path.dirname(filename);
filename = path.basename(filename);
//change directory
process.chdir(directory);
//call grunt
grunt.tasks(['default'], {gruntfile:filename, color:false}, function() {
console.log('done');
});
};
</pre></div><a name="CallingGruntfileFromJavascriptForkModuleArguments"></a><h6>Module Arguments</h6>The module will be called from command line. Node keeps command line arguments inside <code>process.argv</code> array:<br />
<div style="text-align: left;"><pre class="brush:js">module.exports = new function() {
var filename, directory;
this.runGruntfile = function(filename) {
/* ... */
};
//get first command line argument
filename = process.argv[2];
this.runGruntfile(filename);
}();
</pre></div><br />
<a name="CallingGruntfileFromJavascriptForkCallFork"></a><h6>Call Fork</h6>Fork has three arguments: path to module, array with command line arguments and options object. Call <code>module.js</code> with <code>tests/Gruntfile-1.js</code> parameter:<br />
<div style="text-align: left;"><pre class="brush:js">child = cp.fork('./module.js', ['tests/Gruntfile-1.js'], {silent: true})
</pre></div>The <code>silent: true</code> option makes stdout and stderr of the returned <code>child</code> process available inside the parent. If it is set to true, returned object provides access to <code>stdout</code> and <code>stderr</code> streams of the caller. <br />
<br />
Call <code>on('data', callback)</code> on each stream. Passed callback will be called each time the child process sends something to the stream:<br />
<div style="text-align: left;"><pre class="brush:js">child.stdout.on('data', function (data) {
console.log('stdout: ' + data); // handle piece of stdout
});
child.stderr.on('data', function (data) {
console.log('stderr: ' + data); // handle piece of stderr
});
</pre></div><br />
Child process can either crash or end its work correctly:<br />
<div style="text-align: left;"><pre class="brush:js">child.on('error', function(error){
// handle child crash
console.log('error: ' + error);
});
child.on('exit', function (code, signal) {
// this is called after child process ended
console.log('child process exited with code ' + code);
});
</pre></div><br />
Demo project uses following function to calls fork and to bind callbacks:<br />
<div style="text-align: left;"><pre class="brush:js">/**
* callbacks: onProcessError(error), onProcessExit(code, signal), onStdout(data), onStderr(data)
*/
function callGruntfile(filename, callbacks) {
var comArg, options, child;
callbacks = callbacks || {};
child = cp.fork('./test/call-grunt.js', [filename], {silent: true});
if (callbacks.onProcessError) {
child.on("error", callbacks.onProcessError);
}
if (callbacks.onProcessExit) {
child.on("exit", callbacks.onProcessExit);
}
if (callbacks.onStdout) {
child.stdout.on('data', callbacks.onStdout);
}
if (callbacks.onStderr) {
child.stderr.on('data', callbacks.onStderr);
}
}
</pre></div><a name="CallingGruntfileFromJavascriptForkWriteTests"></a><h6>Write Tests</h6>Each unit test calls the <code>callGruntfile</code> function. Callbacks search for expected content inside the standard output stream, check whether exit code was correct, fail when something shows up on error stream or fail if fork call returns an error.<br />
<br />
Success scenario unit test:<br />
<div style="text-align: left;"><pre class="brush:js">pass: function(test) {
var wasPassMessage = false, callbacks;
test.expect(2);
callbacks = {
onProcessError: function(error) {
test.ok(false, "Unexpected error: " + error);
test.done();
},
onProcessExit: function(code, signal) {
test.equal(code, 0, "Exit code should have been 0");
test.ok(wasPassMessage, "Pass message was never sent ");
test.done();
},
onStdout: function(data) {
if (contains(data, 'Plugin worked correctly.')) {
wasPassMessage = true;
}
},
onStderr: function(data) {
test.ok(false, "Stderr should have been empty: " + data);
}
};
callGruntfile('test/gruntfile-pass.js', callbacks);
}
</div></pre>Tests corresponding to failure scenario are pretty much the same and can be found <a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/19176bb0fe680fc0520587791d6f9e89cb0e5183/test/plugin_fork_test.js">on github</a>.<br />
<br />
<a name="CallingGruntfileFromJavascriptForkDisadvantages"></a><h6>Disadvantages</h6>Disadvantages:<ul><li>Used grunt function does not belong to official API.</li>
<li>Child process output streams are available in chunks instead of one big block.</li>
</ul><br />
<a name="CallingGruntfileFromJavascriptSpawn"></a><h5>Spawn</h5><a href="http://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options">Spawn</a> is a cross between fork and exec. Similarly to exec, spawn is able to run an executable file and pass it command line arguments. Child process output streams are treated the same way as in fork. They are send to parent in pieces via callbacks. Therefore, exactly as with fork, using spawn makes sense only if you need arbitrary sized stdout or stderr. <br />
<br />
<a name="CallingGruntfileFromJavascriptSpawnTheProblem"></a><h6>The Problem</h6>The main problem with spawn happens on windows. The name of command to be run must be specified exactly. If you call spawn with an argument <code>grunt</code>, spawn expects executable filename without suffix. Real grunt executable <code>grunt.cmd</code> will not be found. Otherwise said, <code>spawn</code> <a href="https://github.com/joyent/node/issues/2318">ignores windows environment variable PATHEXT</a>.<br />
<br />
<a name="CallingGruntfileFromJavascriptSpawnLoopingSuffixes"></a><h6>Looping Suffixes</h6>If you want to call <code>grunt</code> from <code>spawn</code>, you will need to do one of the following things:<ul><li>use different code for windows and for linux or</li>
<li>read <code>PATHEXT</code> from environment and loop through it until you find the right suffix.</li>
</ul><br />
Following function loops through <code>PATHEXT</code> and passes the right filename to the callback:<br />
<div style="text-align: left;"><pre class="brush:js">function findGruntFilename(callback) {
var command = "grunt", options, extensionsStr, extensions, i, child, onErrorFnc, hasRightExtension = false;
onErrorFnc = function(data) {
if (data.message!=="spawn ENOENT"){
grunt.warn("Unexpected error on spawn " +extensions[i]+ " error: " + data);
}
};
function tryExtension(extension) {
var child = cp.spawn(command + extension, ['--version']);
child.on("error", onErrorFnc);
child.on("exit", function(code, signal) {
hasRightExtension = true;
callback(command + extension);
});
}
extensionsStr = process.env.PATHEXT || '';
extensions = [''].concat(extensionsStr.split(';'));
for (i=0; !hasRightExtension && i<extensions.length;i++) {
tryExtension(extensions[i]);
}
}
</pre></div><a name="CallingGruntfileFromJavascriptSpawnWriteTests"></a><h6>Write Tests</h6>Once you have grunt command name, you are ready to call <code>spawn</code>. Spawn fires exactly the same events as fork, so <code>callGruntfile</code> accepts exactly the same callbacks object and binds its properties to child process events:<br />
<div style="text-align: left;"><pre class="brush:js">function callGruntfile(command, filename, callbacks) {
var comArg, options, child;
callbacks = callbacks || {};
comArg = ["--gruntfile", filename, "--no-color"];
options = {cwd: 'test/'};
child = cp.spawn(command, comArg, options);
if (callbacks.onProcessError) {
child.on("error", callbacks.onProcessError);
}
/* ... callbacks binding exactly as in fork ...*/
}
</pre></div>Tests are also almost the same as those in the previous chapter. Only difference is that you have to find the grunt executable filename before doing everything else. Success scenario test looks like this:<br />
<div style="text-align: left;"><pre class="brush:js">pass: function(test) {
var wasPassMessage = false;
test.expect(2);
findGruntFilename(function(gruntCommand){
var callbacks = {
/* ... callbacks look exactly the same way as in fork ... */
};
callGruntfile(gruntCommand, 'gruntfile-pass.js', callbacks);
});
}
</pre></div>Full success scenario test along with both failure scenarios tests are available <a href="https://github.com/SomMeri/grunt-plugin-testing-demo/blob/19176bb0fe680fc0520587791d6f9e89cb0e5183/test/plugin_spawn_test.js">on github</a>.<br />
<br />
<a name="CallingGruntfileFromJavascriptSpawnDisadvantages"></a><h6>Disadvantages</h6>Disadvantages:<ul><li>Spawn ignores <code>PATHEXT</code> suffixes, custom code to handle it is needed.</li>
<li>Child process output streams are available in chunks instead of one big block.</li>
</ul><br />
<a name="Conclusion"></a><h4>Conclusion</h4>There are three ways how to test grunt plugin from inside gruntfile. Unless you have very strong reason not to, <a href="#CallingGruntfileFromJavascriptExec">use <code>exec</code></a>. <br />
</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com40tag:blogger.com,1999:blog-3355333693246614846.post-40735277862181477302014-09-01T01:00:00.000-07:002015-03-13T10:50:39.787-07:00Mud River Game<div style="text-align: justify;">Mud River is small game I made three months ago. Your goal is to splash as much mud as possible within a short time limit. It was done under a week and can be played on <a href="https://sommeri.github.io/JSPrototype/web/minithings/2 - Mud River/">Github</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://sommeri.github.io/JSPrototype/web/minithings/2 - Mud River/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKa6BX_o2FiLrTyPaAKQEHeWeO81iF9YZRC3w-ev2tT1Or1a7c107IJ6Q57YkAFFUPzN_geS8y_uFO3paAEXV-g-uPIVtEhKlKRJw6Sjy2e8o8ot4s0gGzR6G0o_hgaW81-uJxQE1G3hbl/s320/mud_splash.png" /></a></div><br />
</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com19tag:blogger.com,1999:blog-3355333693246614846.post-32889417081288615302014-08-30T14:24:00.000-07:002014-08-30T14:28:07.915-07:00jUnit: Rules<style>div.blogger-clickTrap {display: none;}</style><style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Rules add special handling around tests, test cases or test suites. They can do additional validations common for all tests in the class, concurrently run multiple test instances, set up resources before each test or test case and tear them down afterwards. <br />
<br />
The rule gets complete control over what will done with the test method, test case or test suite it is applied to. Complete control means that the rule decides what to do before and after running it and how to deal with thrown exceptions.<a name='more'></a><br />
<br />
First chapter shows how to <a href="#UsingRules">use rules</a> and second shows what <a href="#DefaultRules">build-in rules</a> can do. The third chapter describes third party <a href="#ThirdPartyRulesLibraries">rules libraries</a> I found and the last one explains how to <a href="#CustomRule">create new rules</a>.<br />
<br />
<h4>Table Of Contents</h4><ul><li><a href="#UsingRules">Using Rules</a></li>
<ul><li><a href="#UsingRulesExampleRuleTemporaryFolder">Example Rule - Temporary Folder</a></li>
<li><a href="#UsingRulesDeclaringTestRules">Declaring Test Rules</a></li>
<li><a href="#UsingRulesDeclaringClassRules">Declaring Class Rules</a></li>
<li><a href="#UsingRulesUsingRulesInsideTests">Using Rules Inside Tests</a></li>
</ul><li><a href="#DefaultRules">Default Rules</a></li>
<ul><li><a href="#DefaultRulesExpectedException">Expected Exception</a></li>
<li><a href="#DefaultRulesTimeout">Timeout</a></li>
<li><a href="#DefaultRulesErrorCollector">Error Collector</a></li>
<li><a href="#DefaultRulesTestName">Test Name</a></li>
</ul><li><a href="#ThirdPartyRulesLibraries">Third Party Rules Libraries</a></li>
<li><a href="#CustomRule">Custom Rule</a></li>
<ul><li><a href="#CustomRuleNewRule">New Rule</a></li>
<li><a href="#CustomRuleFromScratch">From Scratch</a></li>
<ul><li><a href="#CustomRuleFromScratchFullClass">Full Class</a></li>
</ul><li><a href="#CustomRuleExtendingBuildinClasses">Extending Build-in Classes</a></li>
<ul><li><a href="#CustomRuleExtendingBuildinClassesExternalResource">External Resource</a></li>
<li><a href="#CustomRuleExtendingBuildinClassesVerifier">Verifier</a></li>
</ul></ul><li><a href="#MoreAboutjUnit">More About jUnit</a></li>
</ul><br />
<a name="UsingRules"></a><h4>Using Rules</h4>This chapter shows how to declare and use rules inside a test case. Most rules can be applied to each test method separately, once to the whole test case or once to the whole test suite. Rules run separately for each test are called test rules and rules applied to the whole test case or suite are called class rules. <br />
<br />
We will use temporary folder rule as an example, so first subchapter explains what it does. Second subchapter declares it as test rule and third one as class rule. Last subchapter shows how to access the folder from inside the tests.<br />
<br />
<a name="UsingRulesExampleRuleTemporaryFolder"></a><h5>Example Rule - Temporary Folder</h5>Temporary folder rule creates new empty folder, runs test or test case and then deletes the folder. You can either specify where to create the new folder, or let it be created in system temporary file directory. <br />
<br />
Temporary folder can be used as both test rule and class rule. <br />
<br />
<a name="UsingRulesDeclaringTestRules"></a><h5>Declaring Test Rules</h5>Test rules e.g., rules that run for each test method separately, have to be declared in public field annotated with <code>@Rule</code> annotation.<br />
<br />
Declare test rule:<br />
<div style="text-align: left;"><pre class="brush:java">public class SomeTestCase {
@Rule
public TemporaryFolder folder = new TemporaryFolder();
}
</pre></div><br />
The above <code>folder</code> rule creates new folder before every test method and destroys it afterwards. All tests are able to use that directory, but they are not able to share files through it. Since we used constructor with no parameters, the folder will be created in system temporary file directory. <br />
<br />
Test rule does its work before methods annotated with <code>@Before</code> and after those annotated with <code>@After</code>. Therefore, they will have access to temporary folder too. <br />
<br />
<a name="UsingRulesDeclaringClassRules"></a><h5>Declaring Class Rules</h5>Class rules e.g., rules that run once for the whole test case or test suite, have to be declared in public <i>static</i> field and annotated with <code>@ClassRule</code> annotation.<br />
<br />
Declare test case rule:<br />
<div style="text-align: left;"><pre class="brush:java">public class SomeTestCase {
@ClassRule
public static TemporaryFolder folder = new TemporaryFolder();
}
</pre></div><br />
The above <code>folder</code> rule creates new folder before running the first test method and destroys it after the last one. All tests are able to use that directory and they are able to see files created be previously running tests.<br />
<br />
Class rules are run before anything inside that class. E.g. methods annotated with <code>@BeforeClass</code> or <code>@AfterClass</code> will have access to temporary folder too. The rule runs before and after them.<br />
<br />
<a name="UsingRulesUsingRulesInsideTests"></a><h5>Using Rules Inside Tests</h5>Rules are classes as any other and tests are free to call their public methods and use their public fields. Those calls are used to add test specific configuration to the rule or read data out of it. <br />
<br />
For example, temporary folder can be accessed using <code>newFile</code>, <code>newFolder</code> or <code>getRoot</code> methods. First two create new file or folder inside the temporary folder and the <code>getRoot</code> method returns temporary folder itself.<br />
<br />
Create temporary file and folder:<br />
<div style="text-align: left;"><pre class="brush:java">@Test
public void test1() {
// Create new folder inside temporary directory. Depending on how you
// declared the folder rule, the directory will be deleted either
// right after this test or when the last test in test case finishes.
File file = folder.newFolder("folder");
}
@Test
public void test2() {
// Create new file inside temporary folder. Depending on how you
// declared the folder rule, the file will be deleted either
// right after this test or when the last test in test case finishes.
File file = folder.newFile("file.png");
}
</pre></div><br />
<a name="DefaultRules"></a><h4>Default Rules</h4>JUnit comes with five <a href="https://github.com/junit-team/junit/wiki/Rules">directly useable rules</a>: temporary folder, expected exception, timeout, error collector and test name. Temporary folder have been explained in previous chapter, so we will briefly explain only remaining four rules.<br />
<br />
<a name="DefaultRulesExpectedException"></a><h6>Expected Exception</h6>Expected exception runs the test and catches any exception it throws. The rule is able to check whether the exception contains the right message, the right cause and whether it was thrown by the right line. <br />
<br />
Expected exception has private constructor and must be initialized using static <code>none</code> method. Each exception throwing test has to configure expected exception parameters and then call the <code>expect</code> method of the rule. The rule fails if:<ul><li>the test throws any exception before the <code>expect</code> method call,</li>
<li>the test does not throw an exception after the <code>expect</code> method call,</li>
<li>thrown exception does not have the right message, class or cause.</li>
</ul><br />
The last test line throws an exception. Expected exception rule is configured right before causing the exception:<br />
<div style="text-align: left;"><pre class="brush:java">@Rule
public ExpectedException thrown= ExpectedException.none();
@Test
public void testException() {
// Any exception thrown here causes failure
doTheStuff();
// From now on, the rule expects NullPointerException exception
// to be thrown. If the test finishes without exception or if it
// throws wrong one, the rule will fail.
thrown.expect(NullPointerException.class);
// We well check also message
thrown.expectMessage("Expected Message.");
// this line is supposed to throw exception
theCodeThatThrowsTheException();
}
</pre></div><br />
Bonus: the expected message method accepts also <a href="https://code.google.com/p/hamcrest/wiki/Tutorial">hamcrest</a> matcher argument. That allows you to test the message prefix, suffix, whether it matches some regular expressions or anything else.<br />
<br />
<a name="DefaultRulesTimeout"></a><h6>Timeout</h6>The timeout rule can be used as both test rule and class rule. If it is declared as test rule, it applies the same timeout limit to each test in the class. If it is declared as class rule, it applies the timeout limit to the whole test case or test suite. <br />
<br />
<a name="DefaultRulesErrorCollector"></a><h6>Error Collector</h6>Error collector allows you to run multiple checks inside the test and then report all their failures at once after the test ends. <br />
<br />
Expected-vs-actual value assertions are evaluated using the <code>checkThat</code> method exposed by the rule. It accepts <a href="https://code.google.com/p/hamcrest/wiki/Tutorial">hamcrest</a> matcher as an argument and thus can be used to check anything. <br />
<br />
Unexpected exceptions can be reported directly using <code>addError(Throwable error)</code> method. Alternatively, if you have an instance of <code>Callable</code> to be run, you can call it through <code>checkSucceeds</code> method which adds any thrown exception into errors list.<br />
<br />
<a name="DefaultRulesTestName"></a><h6>Test Name</h6>Test name rule exposes test name inside the test. It might be useful when you need to create custom error reporting.<br />
<br />
<a name="ThirdPartyRulesLibraries"></a><h4>Third Party Rules Libraries</h4>Rules are decoupled from the test class, so it is easy to write libraries of general purpose rules and share them between projects. This chapter describes three such libraries.<br />
<br />
<a href="http://stefanbirkner.github.io/system-rules/">System rules</a> is rules collection for testing code that uses java.lang.System. It is well documented, available in maven and released under Common Public License 1.0 (the same as jUnit). System rules allows you to easily:<br />
<ul><li>test content of <code>System.err</code> and <code>System.out</code>,</li>
<li>simulate input in <code>System.in</code>,</li>
<li>configure system properties and revert their values back,</li>
<li>test <code>System.exit()</code> calls - whether it was called and what return value was,</li>
<li>customize java <code>SecurityManager</code> and revert it back.</li>
</ul><br />
A <a href="https://github.com/aisrael/junit-rules">big set</a> of useful rules is available on aisrael account on github. Its <a href="https://github.com/aisrael/junit-rules/wiki">documentation</a> is somewhat limited, but you can always <a href="https://github.com/aisrael/junit-rules/tree/master/src/main/java/junit/rules">look at the code</a>. All rules are released under MIT license:<br />
<ul><li>starting and stopping in-memory <a href="https://github.com/aisrael/junit-rules/blob/9a20f5b245c3f4e346ea01d571481318b59a19a4/src/main/java/junit/rules/derby/DerbyDataSourceRule.java">derby database</a>,</li>
<li>starting and stopping default java <a href="https://github.com/aisrael/junit-rules/blob/9a20f5b245c3f4e346ea01d571481318b59a19a4/src/main/java/junit/rules/httpserver/HttpServerRule.java">HttpServer</a>,</li>
<li>starting and stopping <a href="https://github.com/aisrael/junit-rules/blob/9a20f5b245c3f4e346ea01d571481318b59a19a4/src/main/java/junit/rules/jetty/JettyServerRule.java">Jetty</a> server,</li>
<li>running <a href="https://github.com/aisrael/junit-rules/blob/9a20f5b245c3f4e346ea01d571481318b59a19a4/src/main/java/junit/rules/jndi/StubJndiContext.java">stub jndi</a>,</li>
<li>some support for <a href="https://github.com/aisrael/junit-rules/tree/master/src/main/java/junit/rules/dbunit">dbUnit</a> tests.</li>
</ul><br />
Another undocumented set of rules <a href="https://github.com/msetkowski/rules">on github</a>. I will not list them here, because their names are self-explanatory and they do not have specified license. Look at the rules <a href="https://github.com/msetkowski/rules/tree/master/src/main/java/org/wowbagger/rules">directory</a> to see their list.<br />
<br />
<a name="CustomRule"></a><h4>Custom Rule</h4>This chapter shows how to create new rules. They can be implemented from scratch by implementing the <code>TestRule</code> interface or by extending one of two convenience classes <code>ExternalResource</code> and <code>Verifier</code> available in jUnit.<br />
<br />
We will create a new rule from scratch and then rewrite it using <code>ExternalResource</code> class.<br />
<br />
<a name="CustomRuleNewRule"></a><h5>New Rule</h5>New rule ensures that all files created by tests are properly deleted after each test finishes its work. The tests themselves have only one responsibility: report all new files using the <code>ensureRemoval(file)</code> method exposed by the rule.<br />
<br />
How to declare and use the <code>DeleteFilesRule</code> rule:<br />
<div style="text-align: left;"><pre class="brush:java">@Rule
public DeleteFilesRule toDelete = new DeleteFilesRule();
@Test
public void example() throws IOException {
// output.css will be deleted whether the test passes, fails or throws an exception
toDelete.ensureRemoval("output.css");
// the compiler is configured to create output.css file
compileFile("input.less");
checkCorrectess("output.css");
}
</pre></div><br />
<a name="CustomRuleFromScratch"></a><h5>From Scratch</h5>Each rule, including class rules, must implement the <code>@TestRule</code> interface. The interface has exactly one method:<br />
<div style="text-align: left;"><pre class="brush:java">public interface TestRule {
Statement apply(Statement base, Description description);
}
</pre></div><br />
Our job is to take statement supplied in the <code>base</code> parameter and turn it into another statement. The statement represents a set of actions e.g., test, test case or test suite to be run. It might have already been modified by other declared rules and includes before and after test or class methods.<br />
<br />
The second <code>description</code> parameter describes the input statement. It can tell test class name, test name, annotations placed on it, it knows whether we are dealing with test or test suite etc. We will not need it.<br />
<br />
We need to create a new statement which will do three things:<ul><li>Empty the list of files to be deleted.</li>
<li>Run underlying test, test case or test suite represented by the <code>base</code> parameter.</li>
<li>Delete all files reported by tests inside previously run statement.</li>
</ul><br />
The statement is a class with one abstract method:<br />
<div style="text-align: left;"><pre class="brush:java">public abstract class Statement {
public abstract void evaluate() throws Throwable;
}
</pre></div><br />
Since underlying statement can throw an exception, the code to delete all files must run from finally block:<br />
<div style="text-align: left;"><pre class="brush:java">public class DeleteFilesRule implements TestRule {
public Statement apply(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
emptyFilesList(); // clean the list of files
try {
base.evaluate(); // run underlying statement
} finally {
removeAll(); // delete all new files
}
}
};
}
}
</pre></div><br />
Both referenced methods <code>emptyFilesList</code> and <code>removeAll</code> are declared outside of new statement, directly inside the <code>DeleteFilesRule</code> class:<br />
<div style="text-align: left;"><pre class="brush:java">public class DeleteFilesRule implements TestRule {
private List<File> toDelete;
private void emptyFilesList() {
toDelete = new ArrayList<File>();
}
private void removeAll() {
for (File file : toDelete) {
if (file.exists())
file.delete();
}
}
/* ... the apply method ... */
}
</pre></div><br />
The last thing we need is a public method able to add files to be deleted:<br />
<div style="text-align: left;"><pre class="brush:java">public void ensureRemoval(String... filenames) {
for (String filename : filenames) {
toDelete.add(new File(filename));
}
}
</pre></div><br />
<a name="CustomRuleFromScratchFullClass"></a><h6>Full Class</h6><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public class DeleteFilesRule implements TestRule {
private List<File> toDelete;
public void ensureRemoval(String... filenames) {
for (String filename : filenames) {
toDelete.add(new File(filename));
}
}
private void emptyFilesList() {
toDelete = new ArrayList<File>();
}
private void removeAll() {
for (File file : toDelete) {
if (file.exists())
file.delete();
}
}
public Statement apply(final Statement base, final Description description) {
return new Statement() {
@Override
public void evaluate() throws Throwable {
emptyFilesList(); // clean the list of files
try {
base.evaluate(); // run underlying statement
} finally {
removeAll(); // delete all new files
}
}
};
}
}
</pre></div><br />
</div></div><a name="CustomRuleExtendingBuildinClasses"></a><h5>Extending Build-in Classes</h5>JUnit contains two convenience classes <code>ExternalResource</code> and <code>Verifier</code> meant to simplify the above process even more. <br />
<br />
<a name="CustomRuleExtendingBuildinClassesExternalResource"></a><h6>External Resource</h6>The <code>ExternalResource</code> helps when you need to do some kind of preprocessing and postprocessing around the underlying test statement. If you need preprocessing, override the <code>before</code> method. If you need postprocessing, override the <code>after</code> method. The <code>after</code> is called from finally block, so it will be run no matter what.<br />
<br />
Our <code>DeleteFilesRule</code> could be rewritten like this:<br />
<div style="text-align: left;"><pre class="brush:java">public class DeleteFilesRule2 extends ExternalResource {
/* ... list, ensureRemoval and removeAll methods ... */
@Override
protected void before() throws Throwable {
toDelete = new ArrayList<File>();
}
@Override
protected void after() {
removeAll();
}
}
</pre></div><br />
<a name="CustomRuleExtendingBuildinClassesVerifier"></a><h6>Verifier</h6>The <code>Verifier</code> has only one method <code>verify</code> to override. That method runs after the wrapped test finished its work and only if it did not thrown an exception. As the name suggests, the verifier is good if you want to run additional checks after the test. <br />
<br />
<a name="MoreAboutjUnit"></a><h4>More About jUnit</h4>Previous post about jUnit 4 features:<br />
<ul><li><a href="http://meri-stuff.blogspot.sk/2014/08/junit-dynamic-tests-generation.html">jUnit: Dynamic Tests Generation</a></li>
</ul></div><br />
<script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>
Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com33tag:blogger.com,1999:blog-3355333693246614846.post-15540643118320137822014-08-23T05:25:00.001-07:002014-08-30T14:30:53.713-07:00jUnit: Dynamic Tests Generation<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Dynamic tests generation is useful when you need to run the same set of tests on many different input values or configurations. It can be achieved either using parametrized tests or using theories. <br />
<br />
Theories are valuable when you have a bunch of data to be used as parameters and want to run tests on all their combinations. You get less control, but you do not have to write combining and iterating code by yourself. Basics about how theories work are explained on <a href="http://www.javacodegeeks.com/2013/12/introduction-to-junit-theories.html">java code geeks</a> (original at <a href="http://www.javaadvent.com/2013/12/introduction-to-junit-theories.html">java advent calendar</a>), so this post focus on parametrized tests.<br />
<br />
Parametrized tests are better when you need to have good control over the input values, e.g. directory with files that are served as an input or a list of meaningful parameters combinations. <a name='more'></a><br />
<br />
<h4>Parametrized Tests</h4>Parametrized test is a test case able to accept parameters and a list of all parameter combinations you want it to run at. JUnit goes through the list of parameters, initializes the test case with each of them and then runs all its test methods.<br />
<br />
Both GUI and Maven runners then interpret each parametrized test run as a separate test. If some of them fails, it is immediately clear which did failed and how many of them failed. <br />
<br />
<h5>Example Use Case</h5>Less4j is less to css compiler, so each of its tests is defined by an input less file and an expected css file. The compiler is run on input file and its output is compared to the expected css. If they match, test is passed. <br />
<br />
All .less files are stored in a directory. Parametrized test case reads that directory and creates one jUnit test for each file. Therefore we can add new tests just by creating new .less and .css, run tests via "run all" button and see new test in all reports.<br />
<br />
<h5>How to Use It</h5>Parametrized test case must have following things:<ul><li>a <code>@RunWith(Parameterized.class)</code> class annotation,</li>
<li>a constructor that accepts test case parameters,</li>
<li>a static method annotated with <code>@Parameters</code> to generate parameters,</li>
<li>test methods that runs on parameters supplied in constructor.</li>
</ul><br />
<h6>Constructor</h6>Parametrized constructor must have at least one parameter. For example, the compiler test case can take input less as a first argument and expected compiled css as second argument. The third argument <code>name</code> is ignored and will be explained later: <br />
<div style="text-align: left;"><pre class="brush:java">@RunWith(Parameterized.class)
public class ParametrizedTest {
public ParametrizedTest(String less, String expectedCss, String name) {
this.less = less;
this.expectedCss = expectedCss;
}
}
</pre></div><br />
<h6>Parameters</h6>The static method generating parameters must return an implementation of the <code>Iterable</code> interface. The iterator returns arrays containing sets of parameters. Each array is used to create one test case instance and objects in it are used as constructor parameters.<br />
<br />
For example, following method returns two arrays and thus leads to two test case instances:<br />
<div style="text-align: left;"><pre class="brush:java">@Parameters(name="Name: {2}")
public static Iterable<Object[]> generateParameters() {
List<Object[]> result = new ArrayList<Object[]>();
result.add(new Object[] {"less", "css", "pass"});
result.add(new Object[] {"less", "error", "fail"});
return result;
}
</pre></div><br />
The <code>name</code> annotation parameter is optional. Its value will be shown in GUI or maven report as the test case name. The <code>{n}</code> is placeholder for n-th array value. They are indexed from 0, so the first test case will be named <code>Name: pass</code> and second test case will be named <code>Name: fail</code>. <br />
<br />
<h6>Test Methods</h6>Parametrized test case can have any number of tests and they must be annotated with <code>@Test</code> annotation:<br />
<div style="text-align: left;"><pre class="brush:java">@Test
public void testCss() { //dummy test method
String actualCss = compile(less);
assertEquals(expectedCss, actualCss);
}
@Test
public void testSourceMap() { //another test method
String actualCss = compile(less);
assertEquals(expectedCss, actualCss);
}
private String compile(String less) { //dummy compile method
return "css";
}
</pre></div><br />
<h6>Output</h6>If you run the above test class, the JUnit view will show following structure:<br />
<div style="text-align: left;"><pre class="brush:ps">[F] com.github.sommeri.jUnit4Examples.ParametrizedTest
[ ] |-- [Name: pass]
[ ] |---------------- testCss[Name: pass]
[ ] |---------------- testSourceMap[Name: pass]
[F] |-- [Name: fail]
[F] |---------------- testCss[Name: fail]
[F] |---------------- testSourceMap[Name: fail]
</pre></div><br />
<h5>Full Test Case</h5><div style="text-align: left;"><pre class="brush:java">@RunWith(Parameterized.class)
public class ParametrizedTest {
private String less;
private String expectedCss;
public ParametrizedTest(String less, String expectedCss, String name) {
this.less = less;
this.expectedCss = expectedCss;
}
@Parameters(name="Name: {2}")
public static Iterable<Object[]> generateParameters() {
List<Object[]> result = new ArrayList<Object[]>();
result.add(new Object[] {"less", "css", "pass"});
result.add(new Object[] {"less", "error", "fail"});
return result;
}
@Test
public void testCss() {
String actualCss = compile(less);
assertEquals(expectedCss, actualCss);
}
@Test
public void testSourceMap() {
String actualCss = compile(less);
assertEquals(expectedCss, actualCss);
}
//dummy compile method
private String compile(String less) {
return "css";
}
}
</pre></div></div><br />
<a name="MoreAboutjUnit"></a><h4>More About jUnit</h4>Following post about jUnit 4 features:<br />
<ul><li><a href="http://meri-stuff.blogspot.sk/2014/08/junit-rules.html">jUnit: Rules</a></li>
</ul><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com35tag:blogger.com,1999:blog-3355333693246614846.post-65858325549593580372014-04-02T01:06:00.001-07:002014-04-02T01:06:37.853-07:00Placeholdered Maps<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Placeholdered map is able to associate keys with values. User can add key value pairs into it and ask for data associated with keys. It works exactly as it would work in an ordinary map.<br />
<br />
The additional feature is ability to use placeholders. Placeholder knows when it was created and is useful to cheat on order in which data were added into the map. If the user adds data into placeholder, the datastructure behaves as if the data were added when the placeholder was created.<br />
<br />
We will create two such structures. <a href="#KeyValueMap">First</a> keeps key value pairs and can return last value associated with the key. Value added into placeholder is returned only if user did not added the same key after the placeholder was created. <a href="#KeyListMap">Second</a> returns all values associated with the key in order they were put in. Placeholders can be used to add data in the middle of those lists.<a name='more'></a><br />
<br />
<h4>Table Of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<ul><li><a href="#FullCode">Full Code</a></li>
<li><a href="#KeyValueMap">Key Value Map</a></li>
<ul><li><a href="#KeyValueMapUsage&TestCases">Usage & Test Cases</a></li>
<ul><li><a href="#KeyValueMapUsage&TestCasesMapLike">Map Like</a></li>
<li><a href="#KeyValueMapUsage&TestCasesPlaceholdersBasics">Placeholders Basics</a></li>
<li><a href="#KeyValueMapUsage&TestCasesPlaceholdersManagement">Placeholders Management</a></li>
<li><a href="#KeyValueMapUsage&TestCasesBulkModifications">Bulk Modifications</a></li>
<li><a href="#KeyValueMapUsage&TestCasesClone">Clone</a></li>
</ul><li><a href="#KeyValueMapSolution">Solution</a></li>
<li><a href="#KeyValueMapImplementation">Implementation</a></li>
<ul><li><a href="#KeyValueMapImplementationLevels">Levels</a></li>
<li><a href="#KeyValueMapImplementationPlaceholderBasics">Placeholder Basics</a></li>
<li><a href="#KeyValueMapImplementationMapLikeMethods">Map Like Methods</a></li>
<li><a href="#KeyValueMapImplementationManagingPlaceholders">Managing Placeholders</a></li>
<li><a href="#KeyValueMapImplementationClone">Clone</a></li>
</ul></ul><li><a href="#KeyListMap">Key List Map</a></li>
<ul><li><a href="#KeyListMapUsage&TestCases">Usage & Test Cases</a></li>
<ul><li><a href="#KeyListMapUsage&TestCasesMapLike">Map Like</a></li>
<li><a href="#KeyListMapUsage&TestCasesPlaceholdersBasics">Placeholders Basics</a></li>
<li><a href="#KeyListMapUsage&TestCasesPlaceholdersManagement">Placeholders Management</a></li>
<li><a href="#KeyListMapUsage&TestCasesBulkModifications">Bulk Modifications</a></li>
<li><a href="#KeyListMapUsage&TestCasesClone">Clone</a></li>
</ul><li><a href="#KeyListMapSolution">Solution</a></li>
<li><a href="#KeyListMapImplementation">Implementation</a></li>
<ul><li><a href="#KeyListMapImplementationLevels">Levels</a></li>
<li><a href="#KeyListMapImplementationPlaceholderBasics">Placeholder Basics</a></li>
<li><a href="#KeyListMapImplementationMapLikeMethods">Map Like Methods</a></li>
<li><a href="#KeyListMapImplementationManagingPlaceholders">Managing Placeholders</a></li>
<li><a href="#KeyListMapImplementationClone">Clone</a></li>
</ul></ul><li><a href="#Discussion">Discussion</a></li>
</ul><br />
</div></div><a name="FullCode"></a><h4>Full Code</h4>Datastructures described in this post are available in a <a href="https://github.com/SomMeri/simplesamples">github repository</a>. <br />
<br />
<a name="KeyValueMap"></a><h4>Key Value Map</h4>Key value store associates keys with values. It is similar to what <code>Map<M, T></code> interface is supposed to do, except that default map does not support placeholders. <br />
<br />
This chapter is split into three parts. <a href="#KeyValueMapUsage&TestCases">First subchapter</a> shows how to use placeholdered key value map. <a href="#KeyValueMapSolution">Second</a> explains how the solution works and <a href="#KeyValueMapImplementation">the last one</a> shows how it is implemented.<br />
<br />
<a name="KeyValueMapUsage&TestCases"></a><h5>Usage & Test Cases</h5>This subchapter shows the most important part of key value map api. Remaining test cases and examples can be found in <a href="https://github.com/SomMeri/simplesamples/blob/master/src/test/java/meristuff/blog/simplesamples/placeholderedkeystorage/KeyValueStorageTest.java"><code>KeyValueStorageTest</code></a> class in our sample project. <br />
<br />
<a name="KeyValueMapUsage&TestCasesMapLike"></a><h6>Map Like</h6>The storage has three basic map like methods <code>add(key, value)</code>, <code>getValue(key)</code> and <code>contains(key)</code>. Add associates keys with values and get returns last added value associated with key in parameter. Contains returns true only if the structure contains given key.<br />
<br />
Otherwise said, the <code>add</code> method overwrites previously added values:<br />
<pre class="brush:java">KeyValueStorage<String, String> storage =
new KeyValueStorage<String, String>();
storage.add("key", "first");
storage.add("key", "second");
assertTrue(storage.contains("key"));
assertEquals("second", storage.getValue("key"));
</pre><br />
<a name="KeyValueMapUsage&TestCasesPlaceholdersBasics"></a><h6>Placeholders Basics</h6>The <code>createPlaceholder</code> method creates new placeholder and the <code>add(placeholder, key, value)</code> method adds data into it.<br />
<br />
Example:<br />
<div style="text-align: left;"><pre class="brush:java">//Create map with placeholder. Placeholder is above last lock value and below last key value.
KeyValueStorage<String, String> storage =
new KeyValueStorage<String, String>();
storage.add("key", "first");
storage.add("lock", "first");
ValuePlaceholder<String, String> placeholder =
storage.createPlaceholder();
storage.add("key", "second");
//add data into placeholder
storage.add(placeholder, "key", "placeholder");
storage.add(placeholder, "lock", "placeholder");
//Last key value is above the placeholder, so placeholder left it unchanged. The value of lock changed, because last lock value is below the placeholder.
assertEquals("second", storage.getValue("key"));
assertEquals("placeholder", storage.getValue("lock"));
</pre></div><br />
<a name="KeyValueMapUsage&TestCasesPlaceholdersManagement"></a><h6>Placeholders Management</h6>Our map supports basic placeholders management. It is customized to be convenient in less4j, so api described in this section may seem little bit arbitrary. Nevertheless, the final datastructure is easy to modify for anyone with different needs.<br />
<br />
A placeholder can be either closed or open and newly created placeholders are considered open. All open placeholders are kept in a list in order they have been created. Structure supports two operations on that list:<ul><li><code>addToFirstPlaceholder</code> adds data into first open placeholder,</li>
<li><code>closeFirstPlaceholder</code> removes first placeholder from open placeholders list.</li>
</ul><br />
Closing only removes the placeholder from open placeholders list. All its data stay in the structure and user still can use it to add additional data. <br />
<br />
Create storage with two placeholders and add data into first one:<br />
<pre class="brush:java">// create structure with two placeholders
KeyValueStorage<String, String> storage =
new KeyValueStorage<String, String>();
storage.add("key", "1");
storage.createPlaceholder();
storage.add("lock", "1");
storage.createPlaceholder();
//add data into first placeholder
storage.addToFirstPlaceholder("key", "placeholder");
storage.addToFirstPlaceholder("lock", "placeholder");
//read data
assertEquals("placeholder", storage.getValue("key"));
assertEquals("1", storage.getValue("lock"));
</pre><br />
Use the <code>closeFirstPlaceholder</code> method to remove first placeholder from open placeholders list. The <code>addToFirstPlaceholder</code> now adds data into second placeholder:<br />
<pre class="brush:java">// close first placeholder - its data are still available
storage.closeFirstPlaceholder();
assertEquals("placeholder", storage.getValue("key"));
//add data into next open placeholder
storage.addToFirstPlaceholder("lock", "second placeholder");
assertEquals("second placeholder", storage.getValue("lock"));
</pre><br />
<a name="KeyValueMapUsage&TestCasesBulkModifications"></a><h6>Bulk Modifications</h6>We needed the ability to replace placeholder with content of another storage. The original placeholder will cease to exist. Data and open placeholders from the other storage will take its place.<br />
<br />
First, create two storages:<br />
<pre class="brush:java">// create storage with one placeholder
KeyValueStorage<String, String> other =
new KeyValueStorage<String, String>();
other.add("under", "other placeholder");
other.createPlaceholder();
other.add("above", "other placeholder");
// create main storage with one placeholder
KeyValueStorage<String, String> target =
new KeyValueStorage<String, String>();
ValuePlaceholder<String, String> holder = target.createPlaceholder();
</pre><br />
Then replace the placeholder in <code>target</code> map by content of the <code>other</code> map. Both data and placeholders are moved into the <code>target</code>:<br />
<pre class="brush:java">//replace target placeholder
target.replacePlaceholder(holder, other);
//data from other storage are availabe inside main
assertEquals("other placeholder", target.getValue("under"));
assertEquals("other placeholder", target.getValue("above"));
//placeholder from other storage is available
target.addToFirstPlaceholder("under", "later on");
target.addToFirstPlaceholder("above", "later on");
assertEquals("later on", target.getValue("under"));
assertEquals("other placeholder", target.getValue("above"));
</pre><br />
<a name="KeyValueMapUsage&TestCasesClone"></a><h6>Clone</h6>Last requirement was especially important. We needed to clone storages. Each clone must have the same data as the original and placeholders on the same positions. Of course, clone and original storage must also be fully independent from each other.<br />
<br />
Create a new storage and clone it:<br />
<pre class="brush:java">// create storage with one placeholder
KeyValueStorage<String, String> original =
new KeyValueStorage<String, String>();
original.add("key", "original");
original.createPlaceholder();
//clone contains the same key value pairs as original storage
KeyValueStorage<String, String> clone = original.clone();
assertEquals("original", clone.getValue("key"));
</pre><br />
Storages are independent. Modification in one is not reflected into another:<br />
<pre class="brush:java">//modifying original storage will not modify clone
original.add("key", "modified");
original.addToFirstPlaceholder("key", "placeholder");
assertEquals("original", clone.getValue("key"));
//clone contain placeholder equivalent to placeholder in original storage
clone.addToFirstPlaceholder("key", "clone has placeholder");
assertEquals("clone has placeholder", clone.getValue("key"));
//modifying it did not changed value in original
assertEquals("placeholder", original.getValue("key"));
</pre><br />
<a name="KeyValueMapSolution"></a><h5>Solution</h5>Data are stored in levels. Each level contains either data added to some placeholder or data added between two placeholders. Levels are kept in list in order they have been created.<br />
<br />
Datastructure with no placeholders has exactly one level of data and simply passes all calls to that level. When user requests new placeholder, storage creates new level of data and associates it with the placeholder. This new level will be reserved for the placeholder. The storage then creates yet another level on top of it and all following add operations put data to the top level. <br />
<br />
Adding data to placeholder is as simple as finding level associated with that placeholder and adding them into it. <br />
<br />
Reading data requires us to loop through levels until we find the one containing the key we are searching for. The search has to start at last added level. If it contains the key, then storage returns found value. If it does not, then we check second last level, third last level and so on, until either value is found or all levels are searched.<br />
<br />
To sum it up, data are kept in levels and those are searched starting with newest and ending with oldest. The storage adds data only:<ul><li>into level associated with some placeholder,</li>
<li>into last level.</li>
</ul><br />
That is it.<br />
<br />
<a name="KeyValueMapImplementation"></a><h5>Implementation</h5>This subchapter shows how the most important key value map methods are implemented. Remaining methods can be found in <a href="https://github.com/SomMeri/simplesamples/blob/master/src/main/java/meristuff/blog/simplesamples/placeholderedkeystorage/KeyValueStorage.java"><code>KeyValueStorage</code></a> class in our sample project. <br />
<br />
<a name="KeyValueMapImplementationLevels"></a><h6>Levels</h6>Each level is essentially an ordinary map hidden behind simplified interface:<br />
<pre class="brush:java">private static class Level<M, T> implements PubliclyCloneable {
private Map<M, T> storage = new HashMap<M, T>();
public void add(M key, T thing) {
storage.put(key, thing);
}
public T getValue(M key) {
return storage.get(key);
}
// ... contains, getAll, toString, etc.
}
</pre><br />
The <code>addAll</code> method copy all key value pairs from another level:<br />
<pre class="brush:java">public void addAll(Level<M, T> otherLevel) {
for (Entry<M, T> entry : otherLevel.storage.entrySet()) {
add(entry.getKey(), entry.getValue());
}
}
</pre><br />
The only slightly interesting method is <code>clone</code>. The default java <code>clone</code> implementation creates shallow clone, so we have to ensure cloned class is independent from original one:<br />
<pre class="brush:java">public Level<M, T> clone() {
try {
@SuppressWarnings("unchecked")
Level<M, T> clone = (Level<M, T>) super.clone();
clone.storage = new HashMap<M, T>(storage);
return clone;
} catch (CloneNotSupportedException e) {
throw new IllegalStateException("Impossible state.");
}
}
</pre><br />
<a name="KeyValueMapImplementationPlaceholderBasics"></a><h6>Placeholder Basics</h6>Each placeholder knows its own level:<br />
<pre class="brush:java">public static class ValuePlaceholder<M, T> {
private final Level<M, T> level;
public ValuePlaceholder(Level<M, T> level) {
super();
this.level = level;
}
}
</pre><br />
Add into placeholder method adds data into level associated with that placeholder:<br />
<pre class="brush:java">public void add(ValuePlaceholder<M, T> placeholder, M key, T thing) {
placeholder.level.add(key, thing);
}
</pre><br />
<a name="KeyValueMapImplementationMapLikeMethods"></a><h6>Map Like Methods</h6>Level are kept in a list:<br />
<pre class="brush:java">public class KeyValueStorage<M, T> {
private LinkedList<Level<M, T>> levels = new LinkedList<Level<M, T>>();
// ...
}
</pre><br />
Add adds data into the last (top) level:<br />
<pre class="brush:java">public void add(M key, T thing) {
Level<M, T> lastLevel = getLastLevel();
lastLevel.add(key, thing);
}
private Level<M, T> getLastLevel() {
if (levels.isEmpty()) {
return addLevel();
}
Level<M, T> lastLevel = levels.peekLast();
return lastLevel;
}
private Level<M, T> addLevel() {
levels.add(new Level<M, T>());
return levels.peekLast();
}
</pre><br />
Both <code>contains</code> and <code>getValue</code> methods loop though list of levels and stop when they find stored key value pair. The <code>getValue</code> method needs to loop through them in reverse:<br />
<pre class="brush:java">public boolean contains(M key) {
for (Level<M, T> level : levels) {
if (level.contains(key))
return true;
}
return false;
}
public T getValue(M key) {
Iterator<Level<M, T>> di = levels.descendingIterator();
while (di.hasNext()) {
Level<M, T> level = di.next();
if (level.contains(key))
return level.getValue(key);
}
return null;
}
</pre><br />
<a name="KeyValueMapImplementationManagingPlaceholders"></a><h6>Managing Placeholders</h6>The storage keeps list of open placeholders and list of levels: <br />
<pre class="brush:java">private LinkedList<ValuePlaceholder<M, T>> placeholders =
new LinkedList<ValuePlaceholder<M, T>>();
private LinkedList<Level<M, T>> levels =
new LinkedList<Level<M, T>>();
</pre><br />
The create placeholder method needs to create two levels. One is associated with placeholder and another is used for data added into storage later on:<br />
<div style="text-align: left;"><pre class="brush:java">public ValuePlaceholder<M, T> createPlaceholder() {
// create level and associate it with new placeholder
ValuePlaceholder<M, T> placeholder =
new ValuePlaceholder<M, T>(addLevel());
placeholders.add(placeholder);
// add level that will be on top of placeholder
addLevel();
return placeholder;
}
private Level<M, T> addLevel() {
levels.add(new Level<M, T>());
return levels.peekLast();
}
</pre></div><br />
Close placeholder method removes first placeholder from the list of open placeholders:<br />
<pre class="brush:java">public void closeFirstPlaceholder() {
placeholders.pop();
}
</pre><br />
Replace placeholder method completely replaces the placeholder by all placehoders from the other storage and its level by all other storage levels:<br />
<div style="text-align: left;"><pre class="brush:java">public void replacePlaceholder(ValuePlaceholder<M, T> placeholder, KeyValueStorage<M, T> otherStorage) {
otherStorage = otherStorage.clone();
//replace in data
replace(levels, placeholder.level, otherStorage.levels);
replace(placeholders, placeholder, otherStorage.placeholders);
}
public static <Q> void replace(List<Q> inList, Q oldElement, List<Q> newElements) {
int level = inList.indexOf(oldElement);
inList.remove(level);
inList.addAll(level, newElements);
}
</pre></div><br />
<a name="KeyValueMapImplementationClone"></a><h6>Clone</h6>Cloning is done in two steps. First, all levels are cloned. Second step creates as many placeholders as the original had. Each has to reference clone of the level original placeholder referenced: <br />
<div style="text-align: left;"><pre class="brush:java">public KeyValueStorage<M, T> clone() {
try {
@SuppressWarnings("unchecked")
KeyValueStorage<M, T> clone = (KeyValueStorage<M, T>) super.clone();
//clone all levels
clone.levels = deeplyClonedLinkedList(levels);
//recreate placeholders - they must reference levels clones
clone.placeholders = new LinkedList<ValuePlaceholder<M, T>>();
for (ValuePlaceholder<M, T> placeholder : placeholders) {
int index = levels.indexOf(placeholder.level);
Level<M, T> levelClone = clone.levels.get(index);
clone.placeholders.add(new ValuePlaceholder<M, T>(levelClone));
}
return clone;
} catch (CloneNotSupportedException e) {
throw new IllegalStateException("Impossible state.");
}
}
public static <T extends PubliclyCloneable> LinkedList<T> deeplyClonedLinkedList(LinkedList<T> list) {
LinkedList<T> result = new LinkedList<T>();
for (T t : list) {
result.add((T)t.clone());
}
return result;
}
</pre></div><br />
<a name="KeyListMap"></a><h4>Key List Map</h4>Key list store associates each key with a list of values. The get method returns all values associated with the key in parameter. Unless placeholders are used, returned list contains values in the same order they were put it. <br />
<br />
The only way to add something in the beginning or the middle of a list is to add it into placeholder. Placeholders know what was in each list when they have been created and add data after last value added below them.<br />
<br />
As before, the chapter is split into three parts. <a href="#KeyListMapUsage&TestCases">First subchapter</a> shows how to use placeholdered key value map. <a href="#KeyListMapSolution">Second</a> explains how the solution works and <a href="#KeyListMapImplementation">the last one</a> shows how it is implemented.<br />
<br />
<br />
<a name="KeyListMapUsage&TestCases"></a><h5>Usage & Test Cases</h5>This subchapter shows the most important part of key value map api. Remaining test cases and examples can be found in <a href="https://github.com/SomMeri/simplesamples/blob/master/src/test/java/meristuff/blog/simplesamples/placeholderedkeystorage/KeyListStorageTest.java"><code>KeyListStorageTest</code></a> class in our sample project. <br />
<br />
<a name="KeyListMapUsage&TestCasesMapLike"></a><h6>Map Like</h6>There are four map-of-lists like methods <code>add(key, value/list)</code>, <code>getValues(key)</code> and <code>contains(key)</code>. Add adds value(s) to the end of list associated with the key in parameter. Get returns list of all values added into the key. Finally, contains returns true only if the structure contains given key.<br />
<br />
Key values list store holds list of values for each key:<br />
<pre class="brush:java">storage.add("key", 1);
storage.add("key", "2");
storage.add("lock", "3");
storage.add("key", "10");
storage.add("lock", "11");
print(storage.get("key")); //prints: 1 2 10
print(storage.get("lock")); //prints: 3 11
</pre><br />
<a name="KeyListMapUsage&TestCasesPlaceholdersBasics"></a><h6>Placeholders Basics</h6>The <code>createPlaceholder</code> method creates new placeholder and the <code>add(placeholder, key, value)</code> method adds data into it.<br />
<br />
Create placeholder in the middle of <code>key</code> list and in the beginning of the <code>lock</code> list:<br />
<pre class="brush:java">storage.add("key", "1");
placeholder = storage.createPlaceholder();
storage.add("key", "2");
storage.add("lock", "3");
//placeholder adds data in the middle of list
storage.add(placeholder, "key", "placeholder");
storage.add(placeholder, "lock", "placeholder");
print(storage.get("key")); //prints: 1 placeholder 2
print(storage.get("lock")); //prints: placeholder 3
</pre><br />
<a name="KeyListMapUsage&TestCasesPlaceholdersManagement"></a><h6>Placeholders Management</h6>Basic placeholders management is exactly the same as in the <a href="#KeyValueMap">key value map</a>. A placeholder can be either closed or open and newly created placeholders are considered open. All open placeholders are kept in a list in order they have been created. Structure supports two operations on that list:<ul><li><code>addToFirstPlaceholder</code> adds data into first open placeholder,</li>
<li><code>closeFirstPlaceholder</code> removes first placeholder from open placeholders list.</li>
</ul><br />
As before, closing only removes the placeholder from open placeholders list. All its data stay in the structure and user still can use it to add additional data. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
Create storage with two placeholders and add data into first one:<br />
<pre class="brush:java">KeyListStorage<String, String> storage =
new KeyListStorage<String, String>();
storage.add("key", "1");
storage.createPlaceholder();
storage.add("lock", "1");
storage.createPlaceholder();
//add data into first placeholder
storage.addToFirstPlaceholder("key", "placeholder");
storage.addToFirstPlaceholder("lock", "placeholder");
//read data
print(storage.getValues("key"));//prints 1 placeholder
print(storage.getValues("lock"));//prints placeholder 1
</pre><br />
Use the <code>closeFirstPlaceholder</code> method to remove first placeholder from open placeholders list. The <code>addToFirstPlaceholder</code> now adds data into second placeholder:<br />
<pre class="brush:java">storage.closeFirstPlaceholder();
print(storage.getValues("key")); //prints 1 placeholder
//add data into next open placeholder
storage.addToFirstPlaceholder("lock", "second placeholder");
print(storage.getValues("lock")); //prints placeholder 1 second placeholder
</pre><br />
</div></div><a name="KeyListMapUsage&TestCasesBulkModifications"></a><h6>Bulk Modifications</h6>User can replace placeholder by content of another storage. The original placeholder will cease to exist. Data and open placeholders from the other storage will take its place.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
First, create two storages:<br />
<div style="text-align: left;"><pre class="brush:java">// create storage with one placeholder
KeyListStorage<String, String> other =
new KeyListStorage<String, String>();
other.add("under", "other placeholder");
other.createPlaceholder();
other.add("above", "other placeholder");
// create main storage with one placeholder
KeyListStorage<String, String> target =
new KeyListStorage<String, String>();
ListPlaceholder<String, String> holder =
target.createPlaceholder();
</pre></div><br />
Then replace the placeholder in <code>target</code> map by content of the <code>other</code> map. Both data and placeholders are moved into the <code>target</code>:<br />
<div style="text-align: left;"><pre class="brush:java">//replace target placeholder
target.replacePlaceholder(holder, other);
//data from other storage are availabe inside main
print(target.getValues("under")); // prints: other placeholder
print(target.getValues("above")); // prints: other placeholder
//placeholder from other storage is available
target.addToFirstPlaceholder("other placeholder", "later on");
target.addToFirstPlaceholder("above", "later on");
print(target.getValues("under")); // prints: other placeholder
print(target.getValues("above")); // prints: later on other placeholder
</pre></div><br />
</div></div><a name="KeyListMapUsage&TestCasesClone"></a><h6>Clone</h6>Finally, clonnig must work exactly the same way as clonning key value maps. Clone must be fully independet from the original object. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
Create new storage and clone it:<br />
<div style="text-align: left;"><pre class="brush:java">// create storage with one placeholder
KeyListStorage<String, String> original = new KeyListStorage<String, String>();
original.createPlaceholder();
original.add("key", "original");
//clone contains the same key value pairs as original storage
KeyListStorage<String, String> clone = original.clone();
assertListsEquals(original.getValues("key"), clone.getValues("key"));
</pre></div><br />
Storages are independent. Modification in one is not reflected into another:<br />
<div style="text-align: left;"><pre class="brush:java">//modifying original storage will not modify clone
original.add("key", "modified");
original.addToFirstPlaceholder("key", "placeholder");
assertOneMemberList("original", clone.getValues("key"));
assertListsEquals(Arrays.asList("placeholder", "original",
"modified"), original.getValues("key"));
//placeholder was cloned too
clone.addToFirstPlaceholder("key", "clone has placeholder");
assertListsEquals(Arrays.asList("clone has placeholder", "original"), clone.getValues("key"));
</pre></div><br />
</div></div><a name="KeyListMapSolution"></a><h5>Solution</h5>Key list map is very similar to key value map shown above. Data are stored in levels. Each level contains either data added to some placeholder or data added between two placeholders. Levels are kept in a list <code>List<Level<M, T>></code>. <br />
<br />
Datastructure with no placeholders has exactly one level of data and simply passes all calls to that level. When user requests new placeholder, storage creates new level of data and associates it with the placeholder. This new level will be reserved for the placeholder. The storage then creates yet another level on top of it and all following <code>add</code> operations put data to the top level. <br />
<br />
Adding data to placeholder is as simple as finding level associated with that placeholder and adding them into it. Reading data requires slightly more, because we have to loop through all levels and put result together from partial lists.<br />
<br />
<a name="KeyListMapImplementation"></a><h5>Implementation</h5>This subchapter shows implementation of the most important key list map methods. Remaining methods can be found in <a href="https://github.com/SomMeri/simplesamples/blob/master/src/main/java/meristuff/blog/simplesamples/placeholderedkeystorage/KeyListStorage.java"><code>KeyListStorage</code></a> class in our sample project. <br />
<br />
Since the implementation of key list map is extremely similar to the key value map, common parts shown in previous chapter are hidden expandable boxes.<br />
<br />
<a name="KeyListMapImplementationLevels"></a><h6>Levels</h6>Levels keep data in <code>Map<M, List<T>></code> and accessed it via <code>getStoredList</code> method:<br />
<div style="text-align: left;"><pre class="brush:java">private Map<M, List<T>> storage = new HashMap<M, List<T>>();
private List<T> getStoredList(M key) {
List<T> list = storage.get(key);
if (list == null) {
list = new ArrayList<T>();
storage.put(key, list);
}
return list;
}
</pre></div><br />
Accessor methods <code>add</code>, <code>get</code> and <code>contains</code> are fairly straightforward and very similar to those in previous key value map.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public void add(M key, T thing) {
getStoredList(key).add(thing);
}
public void add(M key, List<T> things) {
getStoredList(key).addAll(things);
}
public boolean contains(M key) {
return storage.containsKey(key);
}
public List<T> getValues(M key) {
return storage.containsKey(key) ? storage.get(key) :
new ArrayList<T>();
}
// ... addAll, getAllValues, ...
</pre></div><br />
</div></div>The only interesting method is <code>clone</code>. The default java <code>clone</code> implementation creates shallow clone, so we have to ensure cloned level is independent from original one. The map the clone must be filled by copies of original lists:<br />
<div style="text-align: left;"><pre class="brush:java">public Level<M, T> clone() {
try {
@SuppressWarnings("unchecked")
Level<M, T> clone = (Level<M, T>) super.clone();
clone.storage = deeplyClonedMapOfList(storage);
return clone;
} catch (CloneNotSupportedException e) {
throw new IllegalStateException("Impossible state.");
}
}
public static <K, T> Map<K, List<T>> deeplyClonedMapOfList(Map<K, List<T>> map) {
Map<K, List<T>> result = new HashMap<K, List<T>>();
for (Entry<K, List<T>> t : map.entrySet()) {
result.put(t.getKey(), new ArrayList<T>(t.getValue()));
}
return result;
}
</pre></div><br />
<a name="KeyListMapImplementationPlaceholderBasics"></a><h6>Placeholder Basics</h6>Each placeholder keeps reference to its own level and add into placeholder method adds data into it.<br />
<br />
The code is almost the same as in the case of key value map:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public static class ListPlaceholder<M, T> {
private final Level<M, T> level;
public ListPlaceholder(Level<M, T> level) {
this.level = level;
}
}
public void add(ListPlaceholder<M, T> placeholder, M key, List<T> thing) {
placeholder.level.add(key, thing);
}
</pre></div><br />
</div></div><a name="KeyListMapImplementationMapLikeMethods"></a><h6>Map Like Methods</h6>Level are kept in a list:<br />
<pre class="brush:java">LinkedList<Level<M, T>> levels =
new LinkedList<Level<M, T>>();</pre><br />
Add adds data into the last (top) level:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">public void add(M key, T thing) {
Level<M, T> lastLevel = getLastLevel();
lastLevel.add(key, thing);
}
public void add(M key, List<T> thing) {
Level<M, T> lastLevel = getLastLevel();
lastLevel.add(key, thing);
}
</pre><br />
</div></div>The <code>contains</code> method loops though list of levels and stops when it finds stored key list pair:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">public boolean contains(M key) {
for (Level<M, T> level : levels) {
if (level.contains(key))
return true;
}
return false;
}
</pre><br />
</div></div>The <code>getValues</code> method has to go though all levels compose result from partial lists: <br />
<pre class="brush:java">public List<T> getValues(M key) {
LinkedList<T> result = new LinkedList<T>();
for (Level<M, T> level : levels) {
result.addAll(level.getValues(key));
}
return result;
}
</pre><br />
<a name="KeyListMapImplementationManagingPlaceholders"></a><h6>Managing Placeholders</h6>The storage keeps list of open placeholders and list of levels. As before, the create placeholder method needs to create two new levels. One is associated with placeholder and another is used for data added into storage later on. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">private LinkedList<Level<M, T>> levels =
new LinkedList<Level<M, T>>();
private LinkedList<ListPlaceholder<M, T>> placeholders =
new LinkedList<ListPlaceholder<M, T>>();
public ListPlaceholder<M, T> createPlaceholder() {
ListPlaceholder<M, T> placeholder =
new ListPlaceholder<M, T>(addLevel());
placeholders.add(placeholder);
// add level that will be on top of placeholder
addLevel();
return placeholder;
}
private Level<M, T> addLevel() {
levels.add(new Level<M, T>());
return levels.peekLast();
}
</pre></div><br />
</div></div>Close placeholder method removes first placeholder from the list of open placeholders.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public void closeFirstPlaceholder() {
placeholders.pop();
}
</pre></div><br />
</div></div>Replace placeholder method completely replaces the placeholder by all placehoders from the other storage and its level by all other storage levels:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public void replacePlaceholder(ListPlaceholder<M, T> placeholder, KeyListStorage<M, T> otherStorage) {
otherStorage = otherStorage.clone();
//replace in data
replace(levels, placeholder.level, otherStorage.levels);
replace(placeholders, placeholder, otherStorage.placeholders);
}
private <Q> void replace(List<Q> inList, Q oldElement, LinkedList<Q> newElements) {
int indx = inList.indexOf(oldElement);
inList.remove(indx);
inList.addAll(indx, newElements);
}
</pre></div><br />
</div></div><a name="KeyListMapImplementationClone"></a><h6>Clone</h6>Cloning is done in two steps. First, all levels are cloned. Second step creates as many placeholders as the original had. Each has to reference clone of the level original placeholder referenced.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public KeyListStorage<M, T> clone() {
try {
@SuppressWarnings("unchecked")
KeyListStorage<M, T> clone =
(KeyListStorage<M, T>) super.clone();
//clone all levels
clone.levels = deeplyClonedLinkedList(levels);
//recreate placeholders - they must reference levels clones
clone.placeholders = new LinkedList<ListPlaceholder<M, T>>();
for (ListPlaceholder<M, T> placeholder : placeholders) {
int index = levels.indexOf(placeholder.level);
Level<M, T> levelClone = clone.levels.get(index);
clone.placeholders.add(new ListPlaceholder<M, T>(levelClone));
}
return clone;
} catch (CloneNotSupportedException e) {
throw new IllegalStateException("Impossible state.");
}
}
public static <T extends PubliclyCloneable> LinkedList<T> deeplyClonedLinkedList(LinkedList<T> list) {
LinkedList<T> result = new LinkedList<T>();
for (T t : list) {
result.add((T)t.clone());
}
return result;
}
</pre></div><br />
</div></div><a name="Discussion"></a><h4>Discussion</h4>As you can see, the <code>contains</code> and <code>get</code> operations need to go through all levels to find data. That is ok for less4j, because we will not need too many placeholders. There are going to be only few of them.<br />
<br />
If you need a lot of placeholders, another approach with faster get data methods might be needed. Its disadvantage is that it requires more code and is harder to maintain. <br />
<br />
Store all data in one map. No looping is needed inside get data methods, they simply query the map. Each placeholder knows two things for each key:<ul><li>how many elements are stored between it and previous placeholder,</li>
<li>how many elements are stored inside this placeholder.</li>
</ul><br />
Adding key value pair into placeholder differ for both storages. Both have to count all elements stored under the placeholder. Key value map uses that value to find out whether new key value pair should overwrite the one stored in the global map. Key list map uses it as index of where the data should be put in the stored list.<br />
<br />
Initially we used something similar, but we replaced it by solution described in the above post. It is shorter and easier to read.<br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>
<br />
Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com19tag:blogger.com,1999:blog-3355333693246614846.post-66734790059484486302014-03-13T01:17:00.001-07:002014-03-13T03:53:11.905-07:00Falling Ball Game<div style="text-align: justify;">I released simple html5 game called <a href="http://sommeri.github.io/JSPrototype/web/minithings/1%20-%20Falling%20Ball/">Falling Ball</a>. The player uses arrows to move the ball left, right and to make it jump. You get one point for every hit platform and if the ball touches screen border, any of them, then the ball dies.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://sommeri.github.io/JSPrototype/web/minithings/1%20-%20Falling%20Ball/" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUg5He9Cu8NOhguTvfPafMDIJpcwBsvTg9UKDcXKeqmWZfpSas2qYUj6rAhsDrMpLlX_3rcZKIlyBd0l9CNg9zmmeC1SGLV5EtgtMkvt7j3AfP5Lq1h0yhL_v30GCBIW5GnjsjJ7e737Kd/s400/FallingBall-1-80.png" /></a></div><a name='more'></a><br />
<br />
It works in Firefox, Opera and Chrome. Microsoft Explorer shows it in weird colors and does not play sounds, but it still is playable.<!--more--><br />
<br />
<h4>Inspiration</h4>The game was inspired by <a href="www.gamasutra.com/blogs/RamiIsmail/20140226/211807/Game_A_Week_Getting_Experienced_At_Failure.php">game a week challenge</a> and related blog posts by <a href="http://www.gamasutra.com/blogs/AdrielWallick/20140226/211761/Make_Many_Games_Learn_Many_Things.php">Adriel Wallick</a> and <a href="http://gamasutra.com/blogs/ThomasPalef/20140225/211663/What_I_learned_while_doing_my_quotone_game_per_weekquot_challenge.php">Thomas Palef</a>. Thomas Palef made surprisingly enjoyable <a href="http://lessmilk.com/">action/platform games</a> and Adriel Wallick seemed to experiment with <a href="http://www.msminotaur.com/#gameaweek">all kinds of </a>games.<br />
<br />
Thomas Palefs lessmilk games were especially inspiring, since he made games I enjoyed a lot and he was not experienced game developer. They were his first.<br />
<br />
<h4>The Goal</h4>I do not plan to become game developer and I already have projects I'm involved in, so game a week does not sound plausible. However, if they can put together a game every week, I should be able to make a game within a week once in a while. <br />
<br />
My goal was to make html5 game within few days spending on it as little time as possible. No matter how wrong everything will go and how bad or unfinished it would be, the game was supposed to be finished and released within a week.<br />
<br />
However, it had to have all "real game" elements. I wanted it to have sounds, be somewhat polished and somewhat <a href="http://www.youtube.com/watch?v=Fy0aCDmgnxg">juicified</a>. Everything else was supposed to be in <a href="http://onegameamonth.com/faq">one game a month</a> challenge spirit. Anything goes and any cheating is allowed.<br />
<br />
<h4>Development</h4>It was more fun then expected. I did not tried to do proper engineering nor to write unit tests. I did no planning and had no game vision before I started. Instead, I tried to see what is easy to do with the framework I picked up and what kind of assets are available on <a href="http://opengameart.org/">OpenGameArt</a>. <br />
<br />
The nature of the game changed drastically and multiple times during the development. It started as dark platformer, then I attempted to do rhythm game, made some other iterations and ended up with happy action game. <br />
<br />
I used <a href="http://phaser.io/">phaser.io</a> v1.1.3. It was easy to learn and work with. The hardest thing was to choose the right phaser version. Latest 1.1.6 has deal breaking bug - platforms were flying away after being hit by ball. Upcoming 1.2 was just before release, but not there yet. Documentation was not yet fully done and there were only few examples available. <br />
<br />
The version 1.1.3 I picked up is used by tutorials and examples, so it was much easier to learn.<br />
<br />
<h4>Time Spent</h4>The game core is extremely simple, just a few lines of code. It took almost no time to get something running. It is everything else that took most time. <br />
<br />
A lot of it was spend just playing, tweaking constants and seeing whether the game feels right. Then there were many tiny things such as splitting game into levels of right length and giving them names, making you died/you won replay screens as little annoying as possible, counting score and storing it etc. <br />
<br />
Having an easy way to edit levels helped a lot. They need to be tweaked and tested a lot. The amount of time needed to do minor level change directly influences the outcome. <br />
<br />
Finally, digging through creative commons assets was another huge time sink. I spend maybe 30%-40% of time doing that. Fortunately, it was quite relaxing and enjoyable.<br />
<br />
<h4>On Game Design</h4>Turns out that making a game requires you to play it a lot. You become good at it, so it is important to remember how each level felt when you played it for the first time. <br />
<br />
Pictures and sounds make a big difference. The final version of my game have both background music and background image. I tried to save space by removing either of them and the game became way more boring. That was especially surprising in the case of background picture. I thought it is only an optional gimmick. It is not, the game is much less fun without it.<br />
<br />
Overall, juicification is more important then I expected. Too much of it and the player will get distracted and frustrated. Not enough of it and the game is boring.<br />
<br />
<h4>Conclusions</h4>Anyway, I have a <a href="http://sommeri.github.io/JSPrototype/web/minithings/1%20-%20Falling%20Ball/">real game</a> now. Which is improvement over html5 <a href="http://meri-stuff.blogspot.com/2011/11/project-report-jssokoban.html">Sokoban</a> which was only almost there.<br />
</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com4tag:blogger.com,1999:blog-3355333693246614846.post-67502670730981244122014-02-19T15:04:00.000-08:002014-02-19T15:04:25.777-08:00Cryptography & Theory 2: What is Pseudorandom<style>div.blogger-clickTrap {display: none;}</style><style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">As was concluded in the <a href="http://meri-stuff.blogspot.sk/2013/11/cryptography-theory-1-meaning-of-secure.html">first part</a> of this series, security without randomness is impossible. Deterministic ciphers are unable to protect against strong attackers and <a href="http://www.random.org/randomness/">true random</a> generators are impractical or hard to get, so cryptography is build on pseudorandom generators.<br />
<br />
First two chapters of this post define what they are and explain what kind of pseudorandom generators secure cryptography needs. Third chapter introduces yet another way how to talk and think about pseudorandom generators and ciphers in general.<a name='more'></a><br />
<br />
<h4>Pseudorandom Generator Definition</h4>Pseudorandom generator (PRG) is function that generates random looking numbers. It takes small number as an input and generates huge, maybe even infinite, stream of numbers. The input is called a seed.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> Pseudorandom generator (PRG) is an efficient function G:{0, 1}<sup>s</sup> → {0, 1}<sup>n</sup> where s ≪ n (s is much less than n) <br />
<ul><li>{0, 1}<sup>s</sup> - seed space,</li>
<li>{0, 1}<sup>n</sup> - output.</li>
</ul><br />
Even function generating only zeros is pseudorandom generator according to this definition. Of course, using such pseudorandom generator for security is as good as using no pseudorandom generator.<br />
<br />
</div></div><h4>Secure Pseudorandom Generator</h4>Pseudorandom generator good enough to be used for security purposes is called secure pseudorandom generator. Unfortunately, all we can do is to define them. Finding secure pseudorandom generator or proving that a function is secure pseudorandom generator is hard problem. It is not even known whether they exist or not.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
If you can prove that the generator is secure, than P≠NP. <br />
<br />
</div></div>This chapter shows two ways how to define them and proves that they are equivalent to each other. <br />
<br />
<h5>Definition: Unpredictability</h5>Pseudorandom generator is secure if it is unpredictable. If you are able to watch numbers coming out of the generator, then suddenly say "stop, next bit is going to be x" and be right, then you are not watching secure pseudorandom generator. <br />
<br />
The generator counts as predictable if it is possible to guess the next bit with a non-zero accuracy. As usually, the prediction does not have to be correct all the time, it just needs to be better then a coin flip.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> A pseudorandom generator G is predictable if exists an efficient algorithm A and a number i such that:<br />
<ul><li>Pr[A(G(k)|<sub>1,..,i</sub>) = G(k)|<sub>i+1</sub>] ≥ 1/2 + ε for some non-negligible ε.</li>
</ul>where:<br />
<ul><li>|<sub>1,..,i</sub> denotes prefix of length i</li>
<li>|<sub>i+1</sub> denotes i+1th bit.</li>
</ul><br />
<b>Translation:</b> The algorithm A takes first i output bits of the pseudogenerator G as an input and prints its i+1th bit with non zero accuracy.<br />
<br />
<b>Definition:</b> A pseudorandom generator is unpredictable, if it is not predictable. <br />
<br />
</div></div><h5>Definition: Statistical Test and its Advantage</h5>The second way to define secure pseudorandom generator is longer and more complicated. It is also more practical for mathematical proofs constructions, so we have to explain it anyway.<br />
<br />
According to this definition, a pseudorandom generator is secure if it is not possible to write a computer program that would distinguish between its outputs and "truly random" stream of numbers. That computer program would be given a lot of streams and then decide whether they are truly random or generated by the pseudorandom generator.<br />
<br />
<h6>What is Statistical Test</h6>First, we have to define statistical test. Statistical test is an efficient algorithm that takes a stream of numbers as an input and outputs either 0 or 1. It usually checks some property random streams are supposed to have and outputs 1 if input stream looks random.<br />
<br />
For example, a statistical test may compare the number of zeros and the number of ones in its input. If they occur at similar rate, test will output 1.<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> Statistical test is an efficient function A: {0, 1}<sup>n</sup> → {0, 1}. It takes a stream as a parameter and outputs 0 if it is not random and 1 if it is random.<br />
<br />
</div></div>As before, "efficient algorithm" means that it is possible to write corresponding reasonably fast computer program. <br />
<br />
<h6>Using Statistical Test</h6>We will use statistical tests to distinguish between a random stream and a pseudogenerators output. <br />
<br />
Take a lot of random seeds and use them as input for the pseudorandom generator. Run the statistical test on all it outputs. Then take a lot of truly random streams and run the same statistical test on them. <br />
<br />
If the statistical test behaves differently, then it is possible to use that test to distinguish between generators output and truly random streams and the pseudorandom generator is bad.<br />
<br />
<h6>Quality of Statistical Test</h6>We will use something called "the advantage of a statistical test over a pseudorandom generator" to measure how good the statistical test is. The standard way to compute it is to calculate a difference of two probabilities:<ul><li>A probability that the statical test says 1 on the pseudorandom generator output for a random seed k.</li>
<li>A probability that the statistical test says 1 on the truly random stream.</li>
</ul><br />
The higher the advantage is, the better the statistical tests is in distinguishing between random and pseudorandom stream. It is a real number between 0 and 1: <ul><li>1 - distinguish perfectly,</li>
<li>0 - unable to distinguish.</li>
</ul><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> Let G: K → {0, 1}<sup>n</sup> be a pseudorandom generator and A a statistical test on {0, 1}<sup>n</sup>. The advantage of G over A is: <br />
<ul><li>Adv<sub>PRG</sub>[A, G] := |Pr<sub>k →<sup>r</sup> K</sub>[A(G(k))=1] - Pr<sub>r →<sup>r</sup> {0, 1}<sup>n</sup></sub>[A(r)=1]| ∈ [0, 1].</li>
</ul><br />
</div></div><h6>Secure Pseudorandom Generator</h6>Finally, we are ready to define secure pseudorandom generator. A pseudorandom generator is secure, if the advantage of all efficient statistical tests is negligible e.g, very very small. There must be no statistical test that would allow us to distinguish between the output of pseudorandom generator and a truly random stream.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> the pseudorandom generator G: K -> {0,1}<sup>n</sup> is a secure PRG if the advantage of all efficient statistical tests is negligible: <br />
<ul><li>∀ A: Adv<sub>PRG</sub>[A,G] is negligible.</li>
</ul><br />
</div></div><h5>Unpredictability vs Statistical Tests</h5>Both definitions of secure randomness are equal. Any secure pseudorandom generator is unpredictable and any unpredictable pseudorandom generator is secure. <br />
<br />
If you are able to predict the next bit of output, then you are able to create good statistical test. The test would read the beginning of the stream and predict the predictable bit. It would output 1 if the prediction is correct and 0 if it is not. <br />
<br />
As the prediction is somewhat accurate, the statistical test is somewhat able to distinguish between pseudogenerators outputs and truly random streams.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Theorem:</b> A secure pseudorandom generator is unpredictable. <br />
<b>Proof:</b> We prove that if the pseudorandom generator G is predictable, then it is insecure. Suppose that A is an efficient algorithm able to predict some bit of G output:<br />
<ul><li>Pr[A(G(k)|<sub>1..i</sub>) = G(k)|<sub>i+1</sub>] ≤ 1/2+ε for some non-negligible ε.</li>
</ul><br />
We define a statistical test B as a function that runs A and returns 1 iff A was correct:<br />
<ul><li>B(x)=1 iff A(x|<sub>1..i</sub>)=x<sub>i+1</sub></li>
</ul><br />
We will prove that if A can predict the pseudorandom generator G, then B can distinguish it from random and G is not secure.<br />
<br />
The advantage of B is Adv<sub>PRG</sub>[B, G] := |Pr[B(G(k))=1] - Pr[B(r)=1]| = |1/2 - (1/2 + ε)| = |ε| where ε is non-negligible. Therefore the the advantage of B over G is non-negligible. B can distinguish the generator G from random with an advantage epsilon.<br />
∎<br />
<br />
</div></div>The second claim, the one that says unpredictable pseudorandom generator is secure was left as an exercise.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Theorem:</b> An unpredictable pseudorandom generator is secure. <br />
<b>Proof-puzzle:</b> Hint: If the pseudorandom generator G is unpredictable ∀ i ∈ {0, ..., n-1}, then G is a secure pseudorandom generator.<br />
∎<br />
<br />
</div></div><h4>Computationally Indistinguishable</h4>The idea of statistical test can be generalized and used to compare any two streams of numbers. The generalization is what makes the second definition of secure pseudorandom generator useful. It allows us to analyze ciphertexts and other otherwise hard to analyze distribution of numbers.<br />
<br />
<h5>Definition</h5>Two distributions are computationally indistinguishable if it is not possible to write a computer program that would distinguish between them. That computer program would be given a lot of streams and then decide whether they came from one distribution or the other. <br />
<br />
The formal definition is exactly the same as in the previous chapter. Define a statistical test as a function that measures something and outputs 0 or 1. <br />
<br />
Define the advantage of the statistical test as a difference of two probabilities:<ul><li>A probability that the statical test says 1 on a stream generated by the first distribution.</li>
<li>A probability that the statical test says 1 on a stream generated by the second distribution.</li>
</ul><br />
Two distributions are computationally indistinguishable, if the advantage of all efficient statistical tests is negligible. There must be no statistical test that would allow us to distinguish between those two distributions.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> Let P1 and P2 be two distributions over {0, 1}<sup>n</sup>. P1 and P2 are computationally indistinguishable (denoted P1 ≈ P2) if for all statistical tests A:<br />
<ul><li>|Pr<sub>p1 ← P1</sub>[A(p1)=1] - Pr<sub>p2 ← P1</sub>[A(p2)=1]| is negligible.</li>
</ul><br />
</div></div><h5>Yet Another Definition of Secure Pseudorandom Generator</h5>This allows as to create a third equivalent definition of secure pseudorandom generator. A pseudorandom generator is secure, if its outputs are computationally indistinguishable from truly random streams.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> A pseudorandom generator is secure, if {k ←<sup>R</sup>: G(k)} ≈<sub>p</sub> uniform({0, 1}<sup>n</sup>).<br />
<br />
</div></div><h5>Yet Another Definition of Semantically Secure Cipher </h5>We can also redefine a <a href="http://meri-stuff.blogspot.sk/2013/11/cryptography-theory-1-meaning-of-secure.html">semantically secure cipher</a>. The cipher is semantically secure, if the next two sets are computationally indistinguishable:<ul><li>the set of all possible ciphertexts corresponding to one message and</li>
<li>the set of all possible ciphertexts corresponding to any other message.</li>
</ul><br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> The encryption E is semantically secure, if ∀ m<sub>0</sub>, m<sub>1</sub> ∈ M: {E(k, m<sub>0</sub>)} ≈<sub>p</sub> {E(k, m<sub>1</sub>)} e.g. the two distributions are computationally equivalent.<br />
<br />
</div></div>The previous result is very important. It basically says, that no computer program is able to crack ciphertext encrypted with a semantically secure cipher. In other words, if we are willing to change the key often, semantically secure ciphers are all we need.<br />
<br />
<h4>Next ...</h4>Next part will explain how to build ciphers from pseudorandom generators.<br />
</div><br />
<script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com0tag:blogger.com,1999:blog-3355333693246614846.post-90580291093970944562013-11-30T06:47:00.001-08:002014-02-20T02:51:53.867-08:00Cryptography & Theory 1: Meaning of Secure<style>div.blogger-clickTrap {display: none;}</style><style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Cryptography & Theory is series of blog posts on things I learned in coursera <a href="http://meri-stuff.blogspot.sk/2012/06/stanford-free-crypto-class-review.html">stanford online crypto class</a>. The class contained just right mixture of theory, math and programming and I enjoyed it a lot. <br />
<br />
This first part explains what is meant by expression "good cipher". It contains definition of a cipher and multiple definitions of cipher security. Although it does not sounds like much, it is one of the most important things I learned there.<br />
<br />
It is important because ciphers are designed to protect against well defined attacks and are vulnerable to anything else. One can not just take random cipher from build-in ciphers list of his favorite framework and call it a day. One has to understand what is he choosing and why.<a name='more'></a><br />
<br />
<h4>Cipher</h4>A cipher is something that uses secret key to encrypt and decrypt data. Encryption takes key and data as input and produces ciphertext. Decryption takes key and ciphertext and produces data. That is it. <br />
<br />
Ciphers are divided into two big groups: symmetric ciphers and asymmetric ciphers. The difference between them is in how they treat the key. <br />
<br />
<h5>Symmetric Cipher</h5><i>Symmetric</i> cipher uses the same key to encrypt and decrypt data. If you encrypt the message with a key, you have to use the same key do decrypt it. The key must always be kept secret.<br />
<br />
Formal definition of symmetric cipher: <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> A <b>cipher</b> is a pair of efficient algorithms (E, D) defined over a triple (K, M, C) where <br />
<ul><li>E: K×M → C</li>
<li>C: K×C → M</li>
</ul>such that <br />
<ul><li>∀ m ∈ M, k ∈ K: D(k, E(k,m)) = m.</li>
</ul><br />
K in the above definition is set of all possible keys, M is a set of messages and C is a set of all possible ciphertexts. <br />
<br />
The last property is called 'consistency property'. It says that if you encrypt a message, you must be able to decrypt it with the same key.<br />
<br />
</div></div><h5>Asymmetric Cipher</h5><i>Asymmetric</i> cipher uses two different keys. One is used to encrypt data and another is used to decrypt them. It is impossible to use the encryption key to decrypt the ciphertext. Encryption key can be safely published on the internet. <br />
<br />
Asymmetric ciphers usually use symmetric ciphers inside them. This post is about basics behind <i>symmetric</i> ciphers, so there is no need to formally define asymmetric ciphers.<br />
<br />
<h4>Cipher Security</h4>The above definition of cipher includes both good and horrible ciphers. Even cipher that ignores the key and returns the plaintext "as is" satisfies that definition. Therefore, we need to define what is good secure cipher. <br />
<br />
To make things complicated, the meaning of "good cipher" depends a lot on the context. How are you using the cipher and attacker strength are as important as the cipher itself. Can the attacker modify messages? Can you generate new key for every single message or do you have to reuse them? <br />
<br />
The consequence is that there are multiple cipher security definitions. They all model some real world cipher usage and attacker. Some of them have been created after new type of attack has been found and previous definition stopped to be useful.<br />
<br />
<h5>Perfect Secrecy</h5>First cipher security definition was invented by Shannon and defines something called perfect secrecy. The idea of perfect secrecy is that the ciphertext should reveal no information about the original message except its length. No matter how strong the attacker is, ciphertext without a key should be as good as nothing.<br />
<br />
The strongest possible attacker is the one able to try all possible keys. To protect against him, an attempt to try all keys must generate all plaintexts of the same length. If there is a key that decrypts the ciphertext into "yes", then there is another key that decrypts it into "no!" and yet another that decrypts it into "jaj".<br />
<br />
Not only that, each plaintext must appear in the the result exactly the same number of times. If there are two keys that decrypt into "yes", exactly two of them will decrypt it into "no!". <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> A cipher (E, D) over (K, M, C) has <b>perfect secrecy</b> if ∀ c ∈ C and ∀ m<sub>0</sub>, m<sub>1</sub> ∈ M; |m<sub>0</sub>|=|m<sub>1</sub>|: <br />
<ul><li>Pr[E(k,m<sub>0</sub>)=c]=Pr[E(k,m<sub>1</sub>)=c] where k is uniform in K (k←K).</li>
</ul><br />
Note: Pr stands for "probability" and k←K means that the key k is randomly selected. |m<sub>0</sub>|=|m<sub>1</sub>| means that both messages have the same length.<br />
<br />
Any attacker trying to guess which message is encrypted in a ciphertext is as accurate as a coin flip. All messages with the same length have the same chance to be encrypted in the same ciphertext.<br />
<br />
</div></div>Perfect secrecy does not imply perfectly usable cipher. Perfect secrecy has two unfortunate properties:<ul><li>It does not guarantee security if you use the same key twice.</li>
<li>It requires key at least as long as the original message.</li>
</ul><br />
The first property is visible from careful reading of the definition. The definition states only what happen when the attacker has exactly one ciphertext available. It ignores any possibility that he found two ciphertexts encrypted by the same key.<br />
<br />
Proving second property is not that difficult, at least if you know the right trick. Since it is such a nice exercise, I will not spoil it here. The box below states lemma you should try to prove if you take the challenge. It also links pdf with the answer for those who do not want to waste time with silly puzzles.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Theorem:</b> If the cipher (E, D) over (K, M, C) has perfect secrecy, then |K|≥|M| e.g. the number of keys must be bigger than the number of messages.<br />
<b>Proof:</b> <a href="https://wiki.cc.gatech.edu/theory/images/1/1f/Lec2.pdf">available here (pdf)</a> ∎<br />
<br />
</div></div>Therefore, you have to transfer key as long as the message and you have to do it for each secret message. Of course, if you have secure channel able to transfer all those keys, chances are that you could have transferred secret messages through it too. <br />
<br />
The core problem is that the definition of perfect secrecy does not take into account attacker abilities. The definition assumes that attacker knows everything there is to know except key and encrypted message. Real attackers are not that strong and they are not able to try all possible keys.<br />
<br />
<h5>Semantic Security</h5>Semantic security is an attempt to define cipher security in more practical way. As before, the ciphertext should reveal no information about the original message except its length and the attacker can see only one encrypted message.<br />
<br />
However, "reveal" is defined differently. It must be impossible to write reasonably fast computer program able to guess which message is encrypted in ciphertext. "Reasonably fast" rules out attempt to try all possible keys. That would be too slow.<br />
<br />
Formal definition of semantic security involves two player game. One player is named challenger and the other is called adversary. Challenger encrypts messages and adversary is trying to break the cipher. Game rules: <ul><li>Adversary creates two messages of the same length and sends them to the challenger.</li>
<li>Challenger generates random key.</li>
<li>Challenger encrypts one of these messages and sends the cipher to the adversary.</li>
<li>Adversary guesses which message was sent back.</li>
</ul><br />
If the adversary can guess which message was encrypted, then the cipher is not semantically secure. The adversary does not have to be correct all the time, he just must be better then a coin flip. His accuracy is called "semantic security advantage over the cipher under test".<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Definition:</b> Let us describe two experiments EXP(0) and EXP(1). Each experiment EXP(b) requires a challenger and an adversary. The adversary tries to guess which experiment it is in and challenger knows that. They are allowed only following communication:<br />
<br />
<ul><li>Adversary creates two messages m<sub>0</sub> and m<sub>1</sub> of the same length and sends them to the challenger.</li>
<li>Challenger generates random key.</li>
<li>Challenger encrypts the message m<sub>b</sub> and sends E(k,m<sub>b</sub>) to the adversary.</li>
<li>Adversary guesses b.</li>
</ul><br />
Semantic security advantage of the adversary A over the encryption E is<br />
<ul><li>Adv<sub>ss</sub>[A, E] = |Pr[W<sub>0</sub>] - Pr[W<sub>1</sub>]| ∈ [0, 1] where W<sub>b</sub> = [A(EXP(b))=1] e.g. event W<sub>b</sub> happens when adversary is in experiment b and guesses message 1.</li>
</ul><br />
<b>Definition:</b> The encryption E is semantically secure, if for all "efficient" A Adv<sub>ss</sub>[A, E] is negligible.<br />
<br />
Otherwise said, the encryption E is semantically secure, if the adversary's answer is not influenced by the experiment he is in. His chance to say "I'm in experiment 1" is the same in experiment 1 and experiment 0.<br />
<br />
</div></div>Seemingly small difference between "objective probability" and "ability to write computer program" has one important consequence: the key does not have to be as long as the message. However, it still can be used only once and the attacker still gets to see only one encrypted message.<br />
<br />
The attacker in this model knows what might be encrypted in the message. However, he is unable to use that knowledge and extract more information from the ciphertext. Therefore, the definition guarantees something like this: no matter how much the attacker knows about encrypted text, he is not going to learn anything new from the ciphertext.<br />
<br />
Lastly, it is attacker who composes messages. Any semantically secure cipher provides security even if the attacker can compose part of the message you are going to send. It is impossible for him to write his part in a way that would compromise the rest of the message.<br />
<br />
<h5>Chosen Plaintext Attack (CPA)</h5>One time keys are not much practical, so the third definition of secrecy removes one-time-key-usage condition from the semantic security definition. <br />
<br />
Challenger and adversary play the same game as previously. There is only one change: the attacker can compose as many messages as he wants to and see as may ciphertexts as he wants to. Of course, all ciphertexts are encrypted with the same key.<br />
<br />
The adversary and challenger communicate in iterations. At each iteration, the adversary generates two messages m0 and m1 of equal length and sends them to the challenger. The challenger encrypts one of them and sends the cipher back. The challenger has two limitations:<ul><li>It has to use the same key for each iteration.</li>
<li>It has to send back message with the same index at each iteration.</li>
</ul>Adversary has to guess whether challenger sends back messages with index 0 or those with index 1. He does not have to be correct all the time, we only require him to have non-zero accuracy.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Define:</b> Let (E,D) be a cipher over (K,M,C). For b=0,1 define EXP(b) as:<br />
<ul><li>Challenger generates random key.</li>
<li>Adversary queries the challenger any number of times. In each iteration i:<br />
<ul><li>Adversary creates two messages m<sub>i,0</sub> and m<sub>i,1</sub> of the same length and sends them to the challenger.</li>
<li>Challenger encrypts the message m<sub>i,b</sub> and sends E(k, m<sub>i,b</sub>) to the adversary.</li>
</ul></li>
<li>Adversary guesses b.</li>
</ul><br />
<b>Define:</b> A cipher (E,D) over (K,M,C) is secure under the chosen plaintext attack if the advantage of all "efficient" A:<ul><li>Adv<sub>PRF</sub>[A,E]=|Pr[W<sub>0</sub>]-Pr[W<sub>1</sub>]| is negligible. </li>
</ul><br />
Since event W<sub>b</sub> happens when adversary is in experiment b and guesses message 1, the above definition can be rewritten as:<br />
<ul><li>Adv<sub>PRF</sub>[A,E]=|Pr[A(EXP(0))=1]-Pr[A(EXP(1))=1]| is negligible. </li>
</ul><br />
Otherwise said, the encryption E is CPA secure, if the adversary's answer is not influenced by the experiment he is in. His chance to say "I'm in experiment 1" is the same in experiment 1 and experiment 0.<br />
<br />
</div></div>The adversary in chosen plaintext attack definition is so powerful, that the encryption algorithm must be randomized in order to withstand his attacks. If you encrypt the same message twice, you should obtain two different ciphertexts. In short, cpa secure cipher can not be deterministic.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand</a><br />
<div class="answertext" style="text-align: justify;"><br />
<b>Claim:</b> Let (E,D) be a cipher over (K,M,C). If E(k,m) is deterministic, then the cipher is unsecure under the chosen plaintext attack.<br />
<br />
<b>Proof:</b> Adversary needs two queries to the challenger. First, adversary sends the same message twice e.g., m<sub>1,0</sub>=m<sub>1,1</sub>. Challenger encrypts one of these messages and sends back the ciphertext. It does not matter which message is encrypted, the ciphertext is the same in both cases.<br />
<br />
Second query contains one copy of the same same message as before and one different message e.g., m<sub>2,0</sub>=m<sub>1,1</sub> and m<sub>2,1</sub>≠m<sub>1,1</sub>. <br />
<br />
If the challenger sends back same ciphertext as before, then the adversary knows he is in experiment 0. If the challenger sends back different ciphertext as before, then the adversary knows he is in experiment 1. <br />
∎<br />
<br />
</div></div>The fact that it is attacker who composes messages to be encrypted is more important then it might look at the first sight. Some ciphers are perfectly secure against passive attacker, but fail miserably if the attacker can compose messages. <br />
<br />
As before, the attacker is unable to write part of the message in a way that would compromise the rest of it. Plus, no matter how much he knows about encrypted message, he is not going to learn anything new from the ciphertext.<br />
<br />
From all three stated definitions of security, this one is the only suitable for disk encryption. Neither semantically secure cipher not perfectly secure cipher is good enough to encrypt files. At least, if you do not want to change the key after each disk write operation.<br />
<br />
<h5>Practical Security</h5>Last two definitions used terms "efficient" and "reasonably fast" to define adversary abilities. We left them undefined, so here goes one practical estimate: fast enough is faster than 2<sup>90</sup> operations. <br />
<br />
If there is an attack that requires less then 2<sup>90</sup> operations, then the cipher is not secure.<br />
<br />
<h4>Final Warning - Integrity</h4>Strongest cipher security definition explained in this post is "chosen plaintext attack". As far as I know, any cipher that satisfies this definition is able to keep your secrets secret. <br />
<br />
Unfortunately, it still does not mean that such cipher is able to keep your data secure. While the attacker can not extract any new information from the ciphertext, he could be able to <a href="http://meri-stuff.blogspot.sk/2012/04/secure-encryption-in-java.html">modify encrypted text</a>. For example, he still could change your "yes" into "no" or modify account number inside an encrypted bill. <br />
<br />
That brings back the context mentioned in the beginning of this post. A cipher perfectly secure in one context may very easily fail in another. Ciphers are designed to protect against well defined attacks and vulnerable to anything else.<br />
<br />
Of course, there are ways to protect against these kind of attacks too, but that is beyond what I planned to write about. <br />
<br />
<h4>Next ...</h4><a href="http://meri-stuff.blogspot.sk/2014/02/cryptography-theory-2-what-is.html">Next part</a> is about pseudorandom generators. They are important, because all ciphers have to use one if they want to protect against chosen plaintext attack. There is no way around it. <br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>
<br />
Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com2tag:blogger.com,1999:blog-3355333693246614846.post-12559009685504585142013-07-04T07:14:00.000-07:002014-08-15T23:55:06.808-07:00Matasano Crypto Challenge<style>div.blogger-clickTrap {display: none;}</style><div style="text-align: justify;">I recently finished <a href="http://cryptopals.com/">Matasano Crypto Challenges</a> and it was an interesting experience. I started doing them because <a href="https://twitter.com/tqbf">@tqbf</a> tweets with standings showed up in my tweet feed and made me feel competitive. Now I'm very glad I did them.<br />
<br />
<b>Update:</b> competition is not running anymore. All exercises and official solutions are available on <a href="http://cryptopals.com/">cryptopals.org</a>. My solutions are written in java and stored in <a href="https://github.com/SomMeri/matasano-cryptopals-solutions">matasano-cryptopals-solutions</a> github repository.<br />
<br />
Crypto Challenges is a collection of 48 exercises that demonstrate attacks on real world ciphers and protocols. Exercises exploit both badly designed systems and subtle implementation bugs in theoretically rock solid crypto. Most importantly, they make you see how tricky the security can be and how much various details matter. <br />
<br />
If you solved all exercises while the competition was running, Matasano donated 20$ to a charity. <a name='more'></a><br />
<br />
They are all coding exercises and you do not need any prior knowledge of cryptography or security. An exercise may ask you to google and implement cipher, implement vulnerable system using that cipher or finally hack it. None of them was too hard and most have been interesting.<br />
<br />
First sets are more time consuming and harder then later ones. If you already started, do not give up - it will get better.<br />
<br />
It is a bit like trying to do homework if you did not bothered to go to lectures. You have to search the net for algorithms and learn on your own. One exercise even required us to "google for this article and implement attack it describes".<br />
<br />
The most important thing I learned is that exploiting vulnerabilities can be easy. It is much much much easier then I previously thought. I came to conclusion that writing exploits like this should be part of standard CS education. It is easy to underestimate vulnerabilities or consider them theoretical until you see small life like system and suddenly crack it.<br />
<br />
If you have some free time and want to learn a bit of security, you may respond to this challenge. It is worth it.<br />
</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com2tag:blogger.com,1999:blog-3355333693246614846.post-48164928679166761822013-06-20T06:21:00.000-07:002013-07-02T02:52:36.461-07:00Building JavaScript Library with Grunt.js<style>
div.blogger-clickTrap {display: none;}
.greentext .syntaxhighlighter .plain {
color: limegreen !important;
}
</style><style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">New release of typical non trivial JavaScript project needs to run unit tests, concatenate all source files into one and minify the result. Some of them also use code generators, coding style validators or other build time tools. <br />
<br />
Grunt.js is an open source tool able to perform all above steps. It is pluginable and was written in JavaScript, so anyone working on JavaScript library or project should be able to extend it as he needs.<a name='more'></a><br />
<br />
This post explains how to use grunt.js to build JavaScript library. Grunt.js requires node.js and npm to run, so first three chapters explain <a href="#ToolChain">what they are</a>, how to <a href="#Node.jsandNpmInstallation">install them</a> and how to <a href="#NpmBasics">use them</a>. Skip them if you already worked with npm. Grunt configuration is described in <a href="#AddGrunt.jstoProject">fourth</a> and <a href="#ConfigureGrunt.js">fifth</a> chapter. <br />
<br />
Demo library with grunt.js configuration is <a href="https://github.com/SomMeri/grunt-tutorial">avaiable on Github</a>.<br />
<br />
<a name="TableOfContents"></a><h4>Table Of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><ul><li><a href="#TableOfContents">Table Of Contents</a></li>
<li><a href="#ToolChain">Tool Chain Overview</a></li>
<li><a href="#Node.jsandNpmInstallation">Node.js and Npm Installation</a></li>
<ul><li><a href="#Node.jsandNpmInstallationLinux">Linux</a></li>
<li><a href="#Node.jsandNpmInstallationWindows">Windows</a></li>
<li><a href="#Node.jsandNpmInstallationOsX">OsX</a></li>
</ul><li><a href="#NpmBasics">Npm Basics</a></li>
<ul><li><a href="#NpmBasicsOverview">Overview</a></li>
<li><a href="#NpmBasicsGlobalvsLocalInstallation">Global vs Local Installation</a></li>
<li><a href="#NpmBasicsPackage.json">Package.json</a></li>
<li><a href="#NpmBasicsInstallCommand">Install Command</a></li>
</ul><li><a href="#AddGrunt.jstoProject">Add Grunt.js to Project</a></li>
<ul><li><a href="#AddGrunt.jstoProjectOverview">Overview</a></li>
<li><a href="#AddGrunt.jstoProjectInstallation">Installation</a></li>
<li><a href="#AddGrunt.jstoProjectPackage.json">Package.json</a></li>
</ul><li><a href="#ConfigureGrunt.js">Configure Grunt.js</a></li>
<ul><li><a href="#ConfigureGrunt.jsBasicDoNothingConfiguration">Basic Do Nothing Configuration</a></li>
<li><a href="#ConfigureGrunt.jsGruntNpmTasks">Grunt Npm Tasks</a></li>
<ul><li><a href="#ConfigureGrunt.jsGruntNpmTasksInstallthePlugin">Install the Plugin</a></li>
<li><a href="#ConfigureGrunt.jsGruntNpmTasksConfigureTasks">Configure Tasks</a></li>
<li><a href="#ConfigureGrunt.jsGruntNpmTasksLoadandRegisterTasks">Load and Register Tasks</a></li>
</ul><li><a href="#ConfigureGrunt.jsConfigureJSHint">Configure JSHint</a></li>
<ul><li><a href="#ConfigureGrunt.jsConfigureJSHintInstallPlugin">Install Plugin</a></li>
<li><a href="#ConfigureGrunt.jsConfigureJSHintJSHintOptions">JSHint Options</a></li>
<li><a href="#ConfigureGrunt.jsConfigureJSHintConfigureTask">Configure Task</a></li>
<li><a href="#ConfigureGrunt.jsConfigureJSHintLoadandRegister">Load and Register</a></li>
<li><a href="#ConfigureGrunt.jsConfigureJSHintFullJSHintConfiguration">Full JSHint Configuration</a></li>
</ul><li><a href="#ConfigureGrunt.jsConcatenateFiles">Concatenate Files</a></li>
<ul><li><a href="#ConfigureGrunt.jsConcatenateFilesInstallPlugin">Install Plugin</a></li>
<li><a href="#ConfigureGrunt.jsConcatenateFilesLoadPackage.json">Load Package.json</a></li>
<li><a href="#ConfigureGrunt.jsConcatenateFilesComposeBannerandFileName">Compose Banner and File Name</a></li>
<li><a href="#ConfigureGrunt.jsConcatenateFilesConfigureTargetandOptions">Configure Target and Options</a></li>
<li><a href="#ConfigureGrunt.jsConcatenateFilesLoadandRegister">Load and Register</a></li>
<li><a href="#ConfigureGrunt.jsConcatenateFilesFullConcatConfiguration">Full Concat Configuration</a></li>
</ul><li><a href="#ConfigureGrunt.jsMinify">Minify</a></li>
<ul><li><a href="#ConfigureGrunt.jsMinifySourceMaps">Source Maps</a></li>
<li><a href="#ConfigureGrunt.jsMinifyInstallPlugin">Install Plugin</a></li>
<li><a href="#ConfigureGrunt.jsMinifyConfigureTarget">Configure Target</a></li>
<li><a href="#ConfigureGrunt.jsMinifyConfigureOptions">Configure Options</a></li>
<li><a href="#ConfigureGrunt.jsMinifyLoadandRegister">Load and Register</a></li>
<li><a href="#ConfigureGrunt.jsMinifyFullUglifyConfiguration">Full Uglify Configuration</a></li>
</ul><li><a href="#ConfigureGrunt.jsLatestReleaseFile">Last Release File</a></li>
<ul><li><a href="#ConfigureGrunt.jsLatestReleaseFileInstallPlugin">Install Plugin</a></li>
<li><a href="#ConfigureGrunt.jsLatestReleaseFileConfigurePlugin">Configure Plugin</a></li>
</ul><li><a href="#ConfigureGrunt.jsUnitTests">Unit Tests</a></li>
<ul><li><a href="#ConfigureGrunt.jsUnitTestsPrepareTests">Prepare Tests</a></li>
<li><a href="#ConfigureGrunt.jsUnitTestsInstallPlugin">Install Plugin</a></li>
<li><a href="#ConfigureGrunt.jsUnitTestsConfigurePlugin">Configure Plugin</a></li>
</ul><li><a href="#ConfigureGrunt.jsFinalGrunt.jsFile">Final Grunt.js File</a></li>
</ul><li><a href="#End">End</a></li>
</ul></div></div><a name="ToolChain"></a><h4>Tool Chain Overview</h4>We need three tools:<br />
<ul><li><a href="http://nodejs.org">node.js</a>,</li>
<li><a href="http://npmjs.org/">npm</a>,</li>
<li><a href="http://gruntjs.com/">grunt.js</a>.</li>
</ul><br />
Node.js is popular server side JavaScript environment. It is used to write and run JavaScript servers and JavaScript command line tools. If you want to learn more about node.js, this <a href="http://stackoverflow.com/a/5511507">stackoverflow</a> answer links everything you might ever need to get you started.<br />
<br />
Npm is package manager for node.js. It is able to download dependencies from central repository and solve most dependency conflicts. Npm repository stores only node.js server and command line projects. It does not contain libraries meant to be used in web or mobile applications. We will use it to download grunt.<br />
<br />
Grunt.js is task runner we will use to build our project. It is runs on node.js and can be installed from npm.<br />
<br />
<a name="Node.jsandNpmInstallation"></a><h4>Node.js and Npm Installation</h4>Install node.js either directly from node.js <a href="http://nodejs.org/download/">download page</a> or using any of these <a href="https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager">package managers</a>. Successfully installed node.js prints its version number if you type <code>node -v</code> into console.<br />
<br />
Most installers and package managers install both node.js and npm. Type <code>npm -v</code> into console to test whether you have it. If it is available, it will print its version number. What you should do if it is not available depends on your operating system.<br />
<br />
<a name="Node.jsandNpmInstallationLinux"></a><h5>Linux</h5>Download and use <a href=" https://npmjs.org/install.sh">installation script</a>. <br />
<br />
<a name="Node.jsandNpmInstallationWindows"></a><h5>Windows</h5>Windows installer both contains npm and updates path for current user. Separate npm install is needed only if you downloaded node.exe only or compiled it from source.<br />
<br />
Download latest npm zip from <a href="http://nodejs.org/dist/npm/">this page</a>. Unpack and copy it into node.exe installation directory. If you want, you can also update path to have it available anywhere.<br />
<br />
<a name="Node.jsandNpmInstallationOsX"></a><h5>OsX</h5>Npm is bundled inside the installer.<br />
<br />
<a name="NpmBasics"></a><h4>Npm Basics</h4>Understanding of some npm basics is useful for those who need to install and use grunt.js. This is last theoretical chapter and contains only those basics. Anything else can be found in <a href="https://npmjs.org/doc/">npm documentation</a>. <br />
<br />
This chapter explains four things:<ul><li>what is npm,</li>
<li>the difference between local and global npm installation,</li>
<li>what is package.json file and its content,</li>
<li>npm install command.</li>
</ul><br />
<a name="NpmBasicsOverview"></a><h5>Overview</h5>Npm is package manager able to download and install JavaScript dependencies from central repository. Installed packages can be used either as libraries from node.js projects or as command line tools. <br />
<br />
Projects usually keep list of all dependencies inside <code>package.json</code> file and install them from there. Plus, any additional npm library can be installed also from command line. <br />
<br />
<a name="NpmBasicsGlobalvsLocalInstallation"></a><h5>Global vs Local Installation</h5>Each package can be installed either globally or locally. The practical difference is in where they are stored and where they are accessible from.<br />
<br />
<i>Globally installed</i> packages are stored directly inside node.js installation directory. They are called global, because they are available from any directory or node.js project. <br />
<br />
<i>Local installation</i> puts downloaded packages into current working directory. Locally installed packages are then available only from that one directory and its sub-directories.<br />
<br />
Locally installed packages are stored inside <code>node_modules</code> sub-directory. Whatever version control system you use, it is reasonable to add that directory into its <code>.ignore</code> file.<br />
<br />
<a name="NpmBasicsPackage.json"></a><h5>Package.json</h5>Package.json file contains npm project description. It is always located in project root and contains project name, version, license and other similar properties. Most importantly, it contains also two lists with project dependencies.<br />
<br />
First list contains dependencies needed at runtime. Anyone who wish to use the project must install all of them. Second list contains dependencies needed only during the development. Those include test tools, build tools and coding style checkers.<br />
<br />
The easiest way to create it is via <code>npm init</code> command. The command asks few questions and generates basic package.json file in current directory. Only name and version properties are mandatory. If you do not plan to npm publish your library, you can ignore the rest.<br />
<br />
Good package.json descriptions:<ul><li><a href="http://package.json.nodejitsu.com/">package.json interactive guide</a>,</li>
<li><a href="http://docs.nodejitsu.com/articles/getting-started/npm/what-is-the-file-package-json">package.json tutorial on nodejitsu</a>,</li>
<li><a href="https://npmjs.org/doc/json.html">official documentation</a>.</li>
</ul><br />
<a name="NpmBasicsInstallCommand"></a><h5>Install Command</h5>Npm packages are installed using <code>npm install</code> command. The installation is local by default, global installation has to be specified using <code>-g</code> switch.<br />
<br />
The <code>npm install</code> command with no other parameters looks for <code>package.json</code> file in current directory or any of its parent directories. The command then installs all listed dependencies into current directory.<br />
<br />
Concrete npm packages are installed using <code>npm install <pkg_name@version></code> command. The command will find required version of <code>pkg_name</code> package in central repository and install it into current directory. <br />
<br />
Version number <code>@version</code> is optional. If it is missing, npm simply downloads latest available release.<br />
<br />
Finally, the install command invoked with <code>--save-dev</code> switch not only installs the package, but also adds it into package.json as development dependency.<br />
<br />
<a name="AddGrunt.jstoProject"></a><h4>Add Grunt.js to Project</h4>We start grunt configuration by adding Grunt.js into our JavaScript project. We need to install two Grunt.js modules:<ul><li><code>grunt-cli</code> - command line interface (CLI),</li>
<li><code>grunt</code> - task runner.</li>
</ul><br />
Important: grunt.js had backward incompatible release recently. Some older tutorials and documents do not work with last grunt.js version.<br />
<br />
<a name="AddGrunt.jstoProjectOverview"></a><h6>Overview</h6>All real work is done by task runner. Command line interface is only able to parse arguments and to feed them to the task runner. It does nothing useful if the task runner is not installed too.<br />
<br />
Command line interface should be installed globally and task runner locally. Global command line interface ensures that the same grunt commands are available in all directories. Task runner must be local, because various projects may require different grunt versions.<br />
<br />
<a name="AddGrunt.jstoProjectInstallation"></a><h6>Installation</h6>Install global grunt command line interface:<pre class="brush:text">npm install -g grunt-cli</pre><br />
Go to project root and let npm generate package.json file. It will ask few questions and then generate valid package.json file. Only name and version are required, you can ignore the rest.<br />
<pre class="brush:text">npm init</pre><br />
Add latest grunt.js into package.js as development dependency and locally install it at the same time:<br />
<pre class="brush:text">npm install grunt --save-dev</pre><br />
<a name="AddGrunt.jstoProjectPackage.json"></a><h6>Package.json</h6>Package.json created by previous commands looks like this:<br />
<pre class="brush:js">{
"name": "gruntdemo",
"version": "0.0.0",
"description": "Demo project using grunt.js.",
"main": "src/gruntdemo.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": "",
"author": "Meri",
"license": "BSD",
"devDependencies": {
"grunt": "~0.4.1"
}
}
</pre><br />
<a name="ConfigureGrunt.js"></a><h4>Configure Grunt.js</h4>Grunt.js runs tasks and most of its work is done by tasks. However, out of the box grunt.js installation have no tasks available in it. They have to be loaded from plugins and plugins are usually loaded from npm. <br />
<br />
We will use five plugins:<ul><li><a href="#ConfigureGrunt.jsConcatenateFiles">grunt-contrib-concat</a> - concatenates files together,</li>
<li><a href="#ConfigureGrunt.jsMinify">grunt-contrib-uglify</a> - concatenates and minifies files,</li>
<li><a href="#ConfigureGrunt.jsLatestReleaseFile">grunt-contrib-copy</a> - copies files,</li>
<li><a href="#ConfigureGrunt.jsUnitTests">grunt-contrib-qunit</a> - runs unit tests,</li>
<li><a href="#ConfigureGrunt.jsConfigureJSHint">grunt-contrib-jshint</a> - looks for bugs and dangerous constructions in JavaScript code.</li>
</ul><br />
This chapter explains how to configure them. It starts with simplest possible configuration that does nothing and then explains general steps needed to configure a task. Remaining sub-chapters are more practical, each of them explains how to configure one plugin.<br />
<br />
<a name="ConfigureGrunt.jsBasicDoNothingConfiguration"></a><h5>Basic Do Nothing Configuration</h5>Grunt configuration is kept either as JavaScript inside Gruntfile.js file or as CoffeeScript inside Gruntfile.coffe file. Since we are building JavaScript project, we will use the JavaScript version.<br />
<br />
The simplest possible Gruntfile.js file:<br />
<pre class="brush:js">//Wrapper function with one parameter
module.exports = function(grunt) {
// What to do by default. In this case, nothing.
grunt.registerTask('default', []);
};
</pre><br />
The configuration is kept inside <code>module.exports</code> function. It takes <code>grunt</code> object as parameter and configures it by calling its functions.<br />
<br />
Configuration function must create one of more tasks aliases and associate each of them with list of grunt tasks. For example, previous snippet created <code>default</code> tasks alias and associated it with an empty tasks list. In other words, <code>default</code> tasks alias is present, but does nothing.<br />
<br />
Use the <code>grunt <taskAlias></code> command to run all tasks associated with specified <code>taskAlias</code>. The <code>taskAlias</code> argument is optional, grunt will use the <code>default</code> task if it is missing.<br />
<br />
Save Grunt.js file, go to the command line and run grunt:<br />
<pre class="brush:text">grunt</pre><br />
You should see following output:<br />
<div class="greentext"><pre class="brush:text">Done, without errors.</pre></div><br />
Grunt beeps if any configured task returns a warning or an error. If that beeping bothers you, run grunt with <code>-no-color</code> parameter:<br />
<pre class="brush:text">grunt -no-color</pre><br />
<a name="ConfigureGrunt.jsGruntNpmTasks"></a><h5>Grunt Npm Tasks</h5>General steps needed to add task from plugin are the same for all plugins. This chapter shows general overview of what is needed, concrete practical examples are in following chapters.<br />
<br />
<a name="ConfigureGrunt.jsGruntNpmTasksInstallthePlugin"></a><h6>Install the Plugin</h6>First, we need to add the plugin into package.json as development dependency and npm install it:<br />
<pre class="brush:text">npm install <plugin name> --save-dev
</pre><br />
<a name="ConfigureGrunt.jsGruntNpmTasksConfigureTasks"></a><h6>Configure Tasks</h6>Task configuration must be stored in object property named after the task and passed to <code>grunt.initConfig</code> method:<pre class="brush:js">module.exports = function(grunt) {
grunt.initConfig({
firstTask : { /* ... first task configuration ... */ },
secondTask : { /* ... second task configuration ... */ },
// ... all remaining tasks ...
lastTask : { /* ... last task configuration ... */ }
});
// ... the rest ...
};
</pre><br />
Full task configuration possibilities are explained in <a href="http://gruntjs.com/configuring-tasks">grunt.js documentation</a>. This section describes only the most common simple case. It assumes that the task takes a list of files, process them and then generates one output file.<br />
<br />
Simple task configuration:<br />
<pre class="brush:js">firstTask: {
options: {
someOption: value //all this depends on plugin
},
target: {
src: ['src/file1.js', 'src/file2.js'], //input files
dest: 'dist/output.js' // output file
}
}
</pre><br />
Example task configuration has two properties. One contains task options and its name must be <code>options</code>. Grunt.js does not impose any structure on options property, its content depends on used plugin. <br />
<br />
The other can have any name and contains task target. Most common tasks operate on and produce files, so their targets have two properties, <code>src</code> and <code>dest</code>. <code>Src</code> contains list of input files and <code>dest</code> contains output file name.<br />
<br />
If you configure multiple targets, grunt will run the task multiple times - once for each target. Following task will run two times, once for all files in <code>src</code> directory and once for all files in <code>test</code> directory:<br />
<pre class="brush:js">multipleTargetsTask: {
target1: { src: ['src/**/*.js'] },
target2: { src: ['test/**/*.js']] }
}
</pre><br />
<a name="ConfigureGrunt.jsGruntNpmTasksLoadandRegisterTasks"></a><h6>Load and Register Tasks</h6>Finally, tasks from plugins must be loaded with <code>grunt.loadNpmTasks</code> function and registered to some tasks alias. <br />
<br />
All that gives us following Gruntfile.js structure:<pre class="brush:js">module.exports = function(grunt) {
grunt.initConfig({ /* ... tasks configuration ... */ });
grunt.loadNpmTasks('grunt-plugin-name');
grunt.registerTask('default', ['firstTask', 'secondTask', ...]);
};
</pre><br />
<a name="ConfigureGrunt.jsConfigureJSHint"></a><h5>Configure JSHint</h5><a href="http://www.jshint.com/">JSHint</a> detects errors and potential problems in JavaScript code. It was designed to be very configurable and comes with reasonable defaults. <br />
<br />
We will use <a href="https://github.com/gruntjs/grunt-contrib-jshint#readme">grunt-contrib-jshint</a> plugin to integrate it with grunt.js.<br />
<br />
<a name="ConfigureGrunt.jsConfigureJSHintInstallPlugin"></a><h6>Install Plugin</h6>Open console and run <code>npm install grunt-contrib-jshint --save-dev</code> command from project root directory. It will add the plugin into package.json as a development dependency and install it into local npm repository.<br />
<br />
<a name="ConfigureGrunt.jsConfigureJSHintJSHintOptions"></a><h6>JSHint Options</h6>Grunt-contrib-jshint plugin passes all options directly to JSHint. Their complete list is available on JSHint <a href="http://www.jshint.com/docs/">documentation page</a>.<br />
<br />
JSHint option <code>eqeqeq</code> toggles warning on <code>==</code> and <code>!=</code> equality operators. It is turned off by default, because both operators are legitimate. We will turn it on, because they can lead to unexpected results and alternative operators <code>===</code> and <code>!==</code> are safer.<br />
<br />
We will turn on also <code>trailing</code> options which warns about trailing whitespaces in the code. Trailing whitespaces in multi-line strings can cause weird bugs.<br />
<br />
Each option is placed into boolean property with the same name and will be turned on if its value is <code>true</code>. Turn on both <code>eqeqeq</code> and <code>trailing</code> JSHint options:<br />
<pre class="brush:js" style="text-align: left;">options: {
eqeqeq: true,
trailing: true
}
</pre><br />
<a name="ConfigureGrunt.jsConfigureJSHintConfigureTask"></a><h6>Configure Task</h6>Grunt-contrib-jshint plugin has one task named <code>jshint</code>. We will configure it to check all JavaScript files in both src and test directories using options configuration from previous section.<br />
<br />
JSHint configuration must be placed into object property named <code>jshint</code> and sent to <code>grunt.initConfig</code> method. It has two properties, one with options and another with target.<br />
<br />
The target can be placed inside any property other then <code>options</code>, so we will call it simply target. It must contain list of JavaScript files to be checked by JSHint. Files list can be placed inside targets <code>src</code> property and supports both <code>**</code> and <code>*</code> wildcards.<br />
<br />
Validate all JavaScript files in all sub-directores of both <code>src</code> and <code>test</code> directories and turn on two additional JSHint checks:<br />
<pre class="brush:js">grunt.initConfig({
jshint: {
options: {
eqeqeq: true,
trailing: true
},
target: {
src : ['src/**/*.js', 'test/**/*.js']
}
}
});
</pre><br />
<a name="ConfigureGrunt.jsConfigureJSHintLoadandRegister"></a><h6>Load and Register</h6>Final grunt.js part loads grunt-contrib-jshint from npm and registers the task to the default alias:<br />
<pre class="brush:js">grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.registerTask('default', ['jshint']);
</pre><br />
<a name="ConfigureGrunt.jsConfigureJSHintFullJSHintConfiguration"></a><h6>Full JSHint Configuration</h6>Grunt.js with full JSHint configuration:<br />
<pre class="brush:js">module.exports = function(grunt) {
grunt.initConfig({
jshint: {
options: {
trailing: true,
eqeqeq: true
},
target: {
src : ['src/**/*.js', 'test/**/*.js']
}
}
});
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.registerTask('default', ['jshint']);
};
</pre><br />
<a name="ConfigureGrunt.jsConcatenateFiles"></a><h5>Concatenate Files</h5>Our library have to be distributed inside single file whose name contains both version number and project name. That file should start with comment containing library name, version, license, build date and other similar information. <br />
<br />
For example, the version 1.0.2 should be distributed inside file named <code>testGrunt-1.0.2.js</code> and start with following content:<br />
<pre class="brush:js">/*! gruntdemo v1.0.2 - 2013-06-04
* License: BSD */
var gruntdemo = function() {
...
</pre><br />
We will configure the <code>concat</code> task from <a href="https://github.com/gruntjs/grunt-contrib-concat#readme">grunt-contrib-concat</a> plugin to generate the file with right name and initial comment.<br />
<br />
<a name="ConfigureGrunt.jsConcatenateFilesInstallPlugin"></a><h6>Install Plugin</h6>Exactly as before, open console, add the plugin into package.json as a development dependency and install it into local npm repository.<br />
<br />
The command: <code>npm install grunt-contrib-concat --save-dev</code><br />
<br />
<a name="ConfigureGrunt.jsConcatenateFilesLoadPackage.json"></a><h6>Load Package.json</h6>Our first step is to load project information from package.json file and store it in a property. It can be done using <code>grunt.file.readJSON</code> function:<br />
<pre class="brush:js">pkg: grunt.file.readJSON('package.json'),
</pre><br />
Pkg now contains an object corresponding to package.json. Project name is stored inside <code>pkg.name</code> property, version is stored inside <code>pkg.version</code>, license is stored inside <code>pkg.license</code> and so on.<br />
<br />
<a name="ConfigureGrunt.jsConcatenateFilesComposeBannerandFileName"></a><h6>Compose Banner and File Name</h6>Grunt provides <a href="http://gruntjs.com/api/grunt.template">template system</a> we can use to compose banner and file name. Templates are JavaScript expressions embedded into strings using <code><%= expression ></code> syntax. Grunt evaluates the expression and replaces the template with the result. <br />
<br />
For example, <code><%= pkg.name %></code> template is replaced by value of <code>pkg.name</code> property. If the property is string, the template becomes an equivalent of strings concatenation: <code>...' + pkg.name + ' ...</code><br />
<br />
Templates can reference all functions and properties defined in <code>initConfig</code> parameter object and in <code>grunt</code> object. The system also provides few date formatting helper functions. We will use the <code>grunt.template.today(format)</code> function which returns current date in specified format.<br />
<br />
Compose short banner from project name, version, license and current date. Since we will need to reuse the banner in uglify task, we store it in a variable:<br />
<pre class="brush:js">var bannerContent = '/*! <%= pkg.name %> v<%= pkg.version %> - ' +
'<%= grunt.template.today("yyyy-mm-dd") %> \n' +
' * License: <%= pkg.license %> */\n';
</pre><br />
Previous template generates following banner:<br />
<pre class="brush:js">/*! gruntdemo v0.0.1 - 2013-06-04
* License: BSD */</pre><br />
The project name and version part of filename also need to be used in multiple places. Compose file name from project name and version and store it in a variable: <pre class="brush:js">var name = '<%= pkg.name %>-v<%= pkg.version%>';</pre><br />
It generates following name:<br />
<pre class="brush:js">gruntdemo-v0.0.1</pre><br />
<a name="ConfigureGrunt.jsConcatenateFilesConfigureTargetandOptions"></a><h6>Configure Target and Options</h6>The target must contain list of files to be concatenated and name of the file we want to create. Target supports both wildcards and templates, so we can use the template prepared in previous section:<br />
<pre class="brush:js">target : {
// concatenate all files in src directory
src : ['src/**/*.js'],
// place the result into the dist directory,
// name variable contains template prepared in
// previous section
dest : 'distrib/' + name + '.js'
}
</pre><br />
Concat plugin looks for banner in configuration property named <code>banner</code> and <code>banner</code> is the only configuration property we need. Since the banner content is already composed in <code>bannerContent</code> variable, all we need is to place it into configuration property:<br />
<pre class="brush:js">options: {
banner: bannerContent
}
</pre><br />
<a name="ConfigureGrunt.jsConcatenateFilesLoadandRegister"></a><h6>Load and Register</h6>Final part loads grunt-contrib-concat from npm and registers the task to the default alias:<br />
<pre class="brush:js">grunt.loadNpmTasks('grunt-contrib-concat');
grunt.registerTask('default', ['jshint', 'concat']);
</pre><br />
<a name="ConfigureGrunt.jsConcatenateFilesFullConcatConfiguration"></a><h6>Full Concat Configuration</h6>This section shows grunt.js with full concat configuration. <br />
<br />
Note that the <code>pkg</code> property is defined inside the <code>initConfig</code> method parameter. We could not place it elsewhere, because it is accessed from templates and templates have access only to <code>initConfig</code> method parameter and <code>grunt</code> object.<br />
<pre class="brush:js">module.exports = function(grunt) {
var bannerContent = '... banner template ...';
var name = '<%= pkg.name %>-v<%= pkg.version%>';
grunt.initConfig({
// pkg is used from templates and therefore
// MUST be defined inside initConfig object
pkg : grunt.file.readJSON('package.json'),
// concat configuration
concat: {
options: {
banner: bannerContent
},
target : {
src : ['src/**/*.js'],
dest : 'distrib/' + name + '.js'
}
},
jshint: { /* ... jshint configuration ... */ }
});
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-contrib-concat');
grunt.registerTask('default', ['jshint', 'concat']);
};
</pre><br />
<a name="ConfigureGrunt.jsMinify"></a><h5>Minify</h5>Page loading is slower if the browser have to load and parse big files. That may not matter for all projects, but it does matter for mobile apps running on small devices and big web applications using many big libraries.<br />
<br />
Therefore, we are going to produce also minified version of our library. Minification converts input file into smaller one while keeping its functionality unchanged. It removes unimportant whitespaces, shortens constant expressions, gives local variables new shorter names and so on.<br />
<br />
Minification is implemented in <a href="https://github.com/gruntjs/grunt-contrib-uglify#readme">grunt-contrib-uglify</a> plugin which integrates <a href="http://lisperator.net/uglifyjs/">UglifyJs</a> into grunt. Its <code>uglify</code> task concatenates and minifies a set of files.<br />
<br />
<a name="ConfigureGrunt.jsMinifySourceMaps"></a><h6>Source Maps</h6>Minification makes generated files hard to read and very difficult to debug, so we will generate also source map to make these tasks easier.<br />
<br />
Source map links minified file to its source files. If it is available, browser debug tools show original human readable .js files instead of minified version. They are currently supported only by Chrome and nightly builds of Firefox. You can read more about them on <a href="http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/">html5 rocks</a> or <a href="http://net.tutsplus.com/tutorials/tools-and-tips/source-maps-101/">tutplus</a> sites. <br />
<br />
<a name="ConfigureGrunt.jsMinifyInstallPlugin"></a><h6>Install Plugin</h6>Add the plugin into package.json as a development dependency and install it into local npm repository.<br />
<br />
The command: <code>npm install grunt-contrib-uglify --save-dev</code><br />
<br />
<a name="ConfigureGrunt.jsMinifyConfigureTarget"></a><h6>Configure Target</h6>Uglify task target is configured exactly the same way as concat task target. It must contain list of JavaScript files to be minified and name of the file we want to create. <br />
<br />
It support both wildcards and templates, so we can use the template prepared in previous sub-chapter:<br />
<pre class="brush:js">target : {
// use all files in src directory
src : ['src/**/*.js'],
// place the result into the dist directory,
// name variable contains template prepared in
// previous sub-chapter
dest : 'distrib/' + name + '.min.js'
}
</pre><br />
<a name="ConfigureGrunt.jsMinifyConfigureOptions"></a><h6>Configure Options</h6>The banner is configured exactly the same way as in concat - it is read from the <code>banner</code> property and supports templates. Therefore, we can reuse the <code>bannerContent</code> variable prepared in previous sub-chapter.<br />
<br />
<b>Updated</b> A source map is generated only if the <code>sourceMap</code> property is defined. It should contain the name of the source map file. In addition, we have to fill the <code>sourceMapUrl</code> and the <code>sourceMapRoot</code> properties. The first contains relative path from uglified file to the source map file and second contains relative path from source map file to sources.<br />
<br />
Use the <code>bannerContent</code> variable to generate the banner and <code>name</code> variable to generate source map file name:<br />
<pre class="brush:js" style="text-align: left;">options: {
banner: bannerContent,
sourceMapRoot: '../',
sourceMap: 'distrib/'+name+'.min.js.map',
sourceMapUrl: name+'.min.js.map'
}
</pre><br />
<a name="ConfigureGrunt.jsMinifyLoadandRegister"></a><h6>Load and Register</h6>Final part loads grunt-contrib-uglify from npm and registers the task to the default alias:<br />
<pre class="brush:js">grunt.loadNpmTasks('grunt-contrib-uglify');
grunt.registerTask('default', ['jshint', 'concat', 'uglify']);
</pre><br />
<a name="ConfigureGrunt.jsMinifyFullUglifyConfiguration"></a><h6>Full Uglify Configuration</h6>Grunt.js with full uglify configuration: <br />
<pre class="brush:js">module.exports = function(grunt) {
var bannerContent = '... banner template ...';
var name = '<%= pkg.name %>-v<%= pkg.version%>';
grunt.initConfig({
// pkg must be defined inside initConfig object
pkg : grunt.file.readJSON('package.json'),
// uglify configuration
uglify: {
options: {
banner: bannerContent,
sourceMapRoot: '../',
sourceMap: 'distrib/'+name+'.min.js.map',
sourceMapUrl: name+'.min.js.map'
},
target : {
src : ['src/**/*.js'],
dest : 'distrib/' + name + '.min.js'
}
},
concat: { /* ... concat configuration ... */ },
jshint: { /* ... jshint configuration ... */ }
});
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-contrib-concat');
grunt.loadNpmTasks('grunt-contrib-uglify');
grunt.registerTask('default', ['jshint', 'concat', 'uglify']);
};
</pre><br />
<a name="ConfigureGrunt.jsLatestReleaseFile"></a><h5>Last Release File</h5>Last release of our library is now stored in two files and both of them have version number in name. That makes it unnecessary difficult for those who want to automatically download each new version.<br />
<br />
They would be forced to read and parse json file each time they want to find whether new version was released and what its name is. They would also have to update their download scripts if we would decide that we want to change name structure.<br />
<br />
Therefore, we will use the <a href="https://github.com/gruntjs/grunt-contrib-copy#readme">grunt-contrib-copy</a> plugin to create version less copies of all created files.<br />
<br />
<a name="ConfigureGrunt.jsLatestReleaseFileInstallPlugin"></a><h6>Install Plugin</h6>Add the plugin into package.json as a development dependency and install it into local npm repository.<br />
<br />
The command: <code>npm install grunt-contrib-copy --save-dev</code><br />
<br />
<a name="ConfigureGrunt.jsLatestReleaseFileConfigurePlugin"></a><h6>Configure Plugin</h6>Our copy configuration uses three targets, one for each released file. The configuration is without options and basically the same as configuration of previous plugins.<br />
<br />
The only difference is that we need multiple targets. Each target contains src-dest pair with name of file to be copied and name of file to be created.<br />
<br />
We also made one change to previous tasks configuration. We took all file names and placed them into variables, so we can reuse them:<br />
<pre class="brush:js">module.exports = function(grunt) {
/* define filenames */
latest = '<%= pkg.name %>';
name = '<%= pkg.name %>-v<%= pkg.version%>';
devRelease = 'distrib/'+name+'.js';
minRelease = 'distrib/'+name+'.min.js';
sourceMapMin = 'distrib/'+name+'.min.js';
sourceMapUrl = name+'.min.js';
lDevRelease = 'distrib/'+latest+'.js';
lMinRelease = 'distrib/'+latest+'.min.js';
lSourceMapMin = 'distrib/'+latest+'.min.js.map';
grunt.initConfig({
copy: {
development: { // copy non-minified release file
src: devRelease,
dest: lDevRelease
},
minified: { // copy minified release file
src: minRelease,
dest: lMinRelease
},
smMinified: { // source map of minified release file
src: sourceMapMin,
dest: lSourceMapMin
}
},
uglify: { /* ... uglify configuration ... */ },
concat: { /* ... concat configuration ... */ },
jshint: { /* ... jshint configuration ... */ }
});
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-contrib-concat');
grunt.loadNpmTasks('grunt-contrib-uglify');
grunt.loadNpmTasks('grunt-contrib-copy');
grunt.registerTask('default', ['jshint', 'concat', 'uglify', 'copy']);
</pre><br />
<a name="ConfigureGrunt.jsUnitTests"></a><h5>Unit Tests</h5>Finally, we will configure grunt.js to run unit tests on just released files. We will use grunt-contrib-qunit plugin for that purpose. The plugin runs <a href="http://qunitjs.com/">QUnit</a> unit tests in headless <a href="http://phantomjs.org/">PhantomJS</a> instance. <br />
<br />
The solution does not account for browser differences and bugs, but is still good enough. Those who want to have perfect configuration can use <a href="http://code.google.com/p/js-test-driver/">js-test-driver</a> or other similar tool. Js-test-driver configuration is out of this post scope and will not be explained.<br />
<br />
<a name="ConfigureGrunt.jsUnitTestsPrepareTests"></a><h6>Prepare Tests</h6>Qunit unit tests often run over JavaScript files in src directory, because that is practical during the development. If you want to test whether just released concatenated and minified versions work as well, create new QUnit html file and load last release from it.<br />
<br />
Example Qunit entry point file:<br />
<pre class="brush:html"><!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>QUnit Example</title>
<link rel="stylesheet" href="../libs/qunit/qunit.css">
</head>
<body>
<div id="qunit"></div>
<div id="qunit-fixture"></div>
<script src="../libs/qunit/qunit.js"></script>
<!-- Use latest versionless copy of current release -->
<script src="../distrib/gruntdemo.min.js"></script>
<script src="tests.js"></script>
</body>
</html>
</pre><br />
<a name="ConfigureGrunt.jsUnitTestsInstallPlugin"></a><h6>Install Plugin</h6>Add the plugin into package.json as a development dependency and install it into local npm repository.<br />
<br />
The command: <code>npm install grunt-contrib-qunit --save-dev</code><br />
<br />
<a name="ConfigureGrunt.jsUnitTestsConfigurePlugin"></a><h6>Configure Plugin</h6>Grunt-contrib-qunit configuration is the same as the configuration of previous tasks. Since we are fine with the default QUnit configuration, we can omit the options property. All we have to do is to configure the target which must specify all QUnit html files.<br />
<br />
All html files in test directory and its sub-directories should be run as QUnit tests:<br />
<pre class="brush:js">grunt.initConfig({
qunit:{
target: {
src: ['test/**/*.html']
}
},
// ... all previous tasks ...
});
</pre><br />
<a name="ConfigureGrunt.jsFinalGrunt.jsFile"></a><h5>Final Grunt.js File</h5>Click to expand full grunt.js file:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:js">module.exports = function(grunt) {
var name, latest, bannerContent, devRelease, minRelease,
sourceMap, sourceMapUrl, lDevRelease, lMinRelease,
lSourceMapMin;
latest = '<%= pkg.name %>';
name = '<%= pkg.name %>-v<%= pkg.version%>';
bannerContent = '/*! <%= pkg.name %> v<%= pkg.version %> - ' +
'<%= grunt.template.today("yyyy-mm-dd") %> \n' +
' * License: <%= pkg.license %> */\n';
devRelease = 'distrib/'+name+'.js';
minRelease = 'distrib/'+name+'.min.js';
sourceMapMin = 'distrib/'+name+'.min.js.map';
sourceMapUrl = name+'.min.js.map';
lDevRelease = 'distrib/'+latest+'.js';
lMinRelease = 'distrib/'+latest+'.min.js';
lSourceMapMin = 'distrib/'+latest+'.min.js.map';
grunt.initConfig({
pkg: grunt.file.readJSON('package.json'),
qunit:{
target: {
src: ['test/**/*.html']
}
},
// configure copy task
copy: {
development: {
src: devRelease,
dest: lDevRelease
},
minified: {
src: minRelease,
dest: lMinRelease
},
smMinified: {
src: sourceMapMin,
dest: lSourceMapMin
}
},
// configure uglify task
uglify:{
options: {
banner: bannerContent,
sourceMapRoot: '../',
sourceMap: sourceMapMin,
sourceMappingURL: sourceMapUrl
},
target: {
src: ['src/**/*.js'],
dest: minRelease
}
},
// configure concat task
concat: {
options: {
banner: bannerContent
},
target: {
src: ['src/**/*.js'],
dest: devRelease
}
},
// configure jshint task
jshint: {
options: {
trailing: true,
eqeqeq: true
},
target: {
src: ['src/**/*.js', 'test/**/*.js']
}
}
});
grunt.loadNpmTasks('grunt-contrib-jshint');
grunt.loadNpmTasks('grunt-contrib-concat');
grunt.loadNpmTasks('grunt-contrib-uglify');
grunt.loadNpmTasks('grunt-contrib-copy');
grunt.loadNpmTasks('grunt-contrib-qunit');
grunt.registerTask('default', ['jshint', 'concat', 'uglify', 'copy', 'qunit']);
};
</pre><br />
</div></div><a name="End"></a><h4>End</h4>Grunt.js is now configured and ready to be used. Our targets are configured in the most simple possible way, using src-dest pairs, wildcards and templates. Of course, grunt.js provides also other, <a href="http://gruntjs.com/configuring-tasks">more advanced options</a> on how to configure it. <br />
<br />
It would be even better, if it would be possible to automatically download and manage libraries our project depends on. We found two possible solutions, <a href="http://bower.io/">bower</a> and <a href="http://ender.jit.su/">ender</a>. We have not tested them yet, but both should manage front-end JavaScript packages and their dependencies for the web. <br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com4tag:blogger.com,1999:blog-3355333693246614846.post-85651456999119961982013-05-01T07:52:00.000-07:002013-05-08T06:13:27.962-07:00Less CSS - Expressions and Traps<style>div.blogger-clickTrap {display: none;}</style><style type="text/css">div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}</style><div style="text-align: justify;">In an average case, less expressions are easy to use, very useful and behave the same way as expressions in any other programming language. Unfortunately, their interactions with the rest of css syntax can occasionally cause somewhat quirky behavior. Adding to that, next less language version will change in backward incompatible way and expressions are affected too. <br />
<br />
This post shows when and why quirky behavior happens, which problems are going to be solved by planned changes and which traps will still remain there. <a name='more'></a><br />
<br />
It starts with short expressions overview. Less expressions have two differences against what is intuitive and both are explained in second chapter. Third chapter is about backward incompatible changes. There are also two of them and both have major consequences.<br />
<br />
Previous posts in this series:<ul><li><a href="http://meri-stuff.blogspot.sk/2013/03/less-css-tips-and-trics.html">Less CSS - Tips and Trics</a></li>
</ul><br />
<a name="TableofContents"></a><h4>Table of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">[-] Click to Collapse</a><br />
<div class="answertext" style="text-align: justify;"><ul><li><a href="#TableofContents">Table of Contents</a></li>
<li><a href="#Overview">Overview</a></li>
<li><a href="#ExpressionsandAmbiguity">Expressions and Ambiguity</a></li>
<ul><li><a href="#ExpressionsandAmbiguityDivisionvsRation">Division vs Ration</a></li>
<ul><li><a href="#ExpressionsandAmbiguityDivisionvsRationLess1.3.xWay">Less 1.3.x Way</a></li>
<li><a href="#ExpressionsandAmbiguityDivisionvsRationLess1.4.xWay">Less 1.4.x Way</a></li>
</ul><li><a href="#ExpressionsandAmbiguityListsvsMathOperations">Lists vs Math Operations</a></li>
<ul><li><a href="#ExpressionsandAmbiguityListsvsMathOperationsBreakdown">Breakdown</a></li>
<li><a href="#ExpressionsandAmbiguityListsvsMathOperationsAmbiguitySolution">Ambiguity Solution</a></li>
<li><a href="#ExpressionsandAmbiguityListsvsMathOperationsLess1.3.xDetails">Less 1.3.x Details</a></li>
<li><a href="#ExpressionsandAmbiguityListsvsMathOperationsLess1.4.xDetails">Less 1.4.x Details</a></li>
</ul></ul><li><a href="#BackwardIncompatibleChanges">Backward Incompatible Changes</a></li>
<ul><li><a href="#BackwardIncompatibleChangesRequiredParentheses">Required Parentheses</a></li>
<ul><li><a href="#BackwardIncompatibleChangesRequiredParenthesesBasicExamples">Basic Examples</a></li>
<li><a href="#BackwardIncompatibleChangesRequiredParenthesesMixins">Mixins</a></li>
<li><a href="#BackwardIncompatibleChangesRequiredParenthesesVariables">Variables</a></li>
<li><a href="#BackwardIncompatibleChangesRequiredParenthesesFunctions">Functions</a></li>
</ul><li><a href="#BackwardIncompatibleChangesExpressionsandUnits">Expressions and Units</a></li>
<ul><li><a href="#BackwardIncompatibleChangesExpressionsandUnitsUnitConversionsforPlusandMinus">Unit Conversions for Plus and Minus</a></li>
<li><a href="#BackwardIncompatibleChangesExpressionsandUnitsUnitsExpressionsSimplificationforDivisionandMultiplication">Notes on Division and Multiplication</a></li>
</ul></ul><li><a href="#Preview">Preview Availability</a></li>
<li><a href="#Conclusion">Conclusion</a></li>
</ul><br />
</div></div><h4>Warning</h4>Some aspects of planned backward compatible changes are still under discussion. This post describes proposed changes as implemented in third beta version of less-1.4.0.js. <br />
<br />
Since the discussion started after this post was written, I decided to release the post anyway and keep it as is. People interested in upcoming changes may still find it useful and it may take some time until all discussions are concluded.<br />
<br />
List of active discussions will be updated as new issues are opened and closed:<ul><li><a href="https://github.com/cloudhead/less.js/issues/1259">discussion</a> related to <a href="#BackwardIncompatibleChangesRequiredParentheses">required parentheses</a> change described in this post.</li>
</ul><br />
<b>Update:</b> The discussion have been concluded in the meantime. The required parentheses backward incompatible change will be postponed. Next less.js release 1.4.0 will not have it. The change will be released in less 2.0. The rest of post have been updated.<br />
<br />
<a name="Overview"></a><h4>Overview</h4>Expressions are used to perform basic mathematical operations directly inside less style sheets. They can evaluate four basic arithmetical operators (<code>+</code>, <code>-</code>, <code>*</code> and <code>/</code>), know how to handle parentheses <code>()</code>, can reference variables <code>@variable-name</code> and even support several <a href="http://lesscss.org/#reference">build-in functions</a>.<br />
<br />
Example:<br />
<pre class="brush:css">@variable: 3;
#selector {
arithmetical-operators: (3*10+5);
parentheses: (3*(10+5));
reference-variable: (@variable-2);
/* note: sine function will be available in upcoming less.js 1.4.0 */
built-in-function-sine: sin(1);
}
</pre><br />
compiles into:<pre class="brush:css">#selector {
arithmetical-operators: 35;
parentheses: 45;
reference-variable: 1;
built-in-function-sine: 0.8414709848078965;
}
</pre><br />
<a name="ExpressionsandAmbiguity"></a><h4>Expressions and Ambiguity</h4>As we wrote in the overview, interactions between expressions and css syntax lead to two less language quirks. While both are easy to deal with, both may be also surprising for someone who run into them for the first time.<br />
<br />
Ambiguous situations:<ul><li><a href="#ExpressionsandAmbiguityDivisionvsRation">double meaning of slash <code>/</code> symbol</a>,</li>
<li><a href="#ExpressionsandAmbiguityListsvsMathOperations">double meaning of space</a>.</li>
</ul><br />
Problems with <a href="#ExpressionsandAmbiguityDivisionvsRation">double meaning of slash <code>/</code> symbol</a> occurred only rarely. Unfortunately, they have been impossible to solve and thus triggered the biggest backward incompatible change less 2.0.0 will have. The second issue, <a href="#ExpressionsandAmbiguityListsvsMathOperations">double meaning of space</a>, is something almost every less user will run into.<br />
<br />
<a name="ExpressionsandAmbiguityDivisionvsRation"></a><h5>Division vs Ration</h5>Less uses slash <code>/</code> to express division and css uses it to denote ratio in fonts and media queries. Look at following less style:<pre class="brush:css">@baseLineHeight: 10;
@media (aspect-ratio: 16/10){
div {
margin-top: (@baseLineHeight/2);
}
}
</pre><br />
The above less contains two slashes:<ul><li>media query: <code>aspect-ratio: 16/10</code>,</li>
<li>declaration: <code>margin-top: (@baseLineHeight/2)</code>.</li>
</ul><br />
The first slash is media query ratio and should compile into <code>aspect-ratio: 16/10</code> while the second slash is division and should compile into <code>margin-top: 5</code>. The whole less should compile into following css:<pre class="brush:css">@media (aspect-ratio: 16/10){
div {
margin-top: 5;
}
}
</pre><br />
The problem: symbol slash <code>/</code> is ambiguous and means different things in different contexts. This ambiguity is solved differently by 1.3.x and 2.0.0 versions. <br />
<br />
Note: this change was originally planned for less.js 1.4.0. It was postponed to give users more time before the change.<br />
<br />
<a name="ExpressionsandAmbiguityDivisionvsRationLess1.3.xWay"></a><h6>Less 1.3.x Way</h6>Ratios are used mainly in media queries and fonts, so expressions evaluation in media queries and fonts have been turned off. This prevents misinterpreted ratio slashes in correct css, but has one huge obvious drawback - you can not use expressions in them.<br />
<br />
Any slash in font declaration or media query is treated as ratio and will be unmodified:<br />
<pre class="brush:css">#selector {
font: 4/2;
}
@media (aspect-ratio: 16/9) { ... }
</pre><br />
compiles into:<br />
<pre class="brush:css">#selector {
font: 4/2;
}
@media (aspect-ratio: 16/9) { ... }
</pre><br />
Expressions in font declaration or media query are not supported and the outcome is either syntax error or undefined:<br />
<pre class="brush:css">#selector {
/* expression not evaluated: undefined result */
font: 4+2;
}
/* expression not evaluated: syntax error */
@media (aspect-ratio: (16/9)) { ... }
/* expression not evaluated: syntax error */
@media (min-width: 2+3) { ... }
</pre><br />
Expressions on any other place are treated as math operations and evaluated. All slashes other then those in fonts and media queries are treated as division:<br />
<pre class="brush:css">#selector {
margin: 4/2;
}
</pre><br />
compiles into:<br />
<pre class="brush:css">#selector {
margin: 2;
}
</pre><br />
<a name="ExpressionsandAmbiguityDivisionvsRationLess1.4.xWay"></a><h6>Less 2.x.x Way</h6>Less 2.0.0 changed expression evaluation to work the same way everywhere. There will be no difference between media queries, fonts and everything else. <br />
<br />
Instead, the expression must be enclosed into parentheses in order to be evaluated. If it is not inside parentheses, then it is going to be copied into output as is. Details are described in <a href="#BackwardIncompatibleChanges">backward incompatible changes</a> chapter, so this section shows only few simple examples.<br />
<br />
Expressions in fonts and other properties are evaluated exactly the same way:<br />
<pre class="brush:css">#fonts {
font: 4/2; // no evaluation: compiles into "font: 4/2"
font: 4+2; // no evaluation: compiles into "font: 4+2"
font: (4/2); // evaluation: compiles into "font: 2"
font: (4+2); // evaluation: compiles into "font: 6"
}
#margins {
margin: 4/2; // no evaluation: compiles into "margin: 4/2"
margin: 4+2; // no evaluation: compiles into "margin: 4+2"
margin: (4/2); // evaluation: compiles into "margin: 2"
margin: (4+2); // evaluation: compiles into "margin: 6"
}</pre><br />
Media queries evaluation is also the same:<br />
<pre class="brush:css">// no evaluation: compiles into "aspect-ratio: 16/9"
@media (aspect-ratio: 16/9) {
h1{ color:green;}
}
// evaluation: compiles into "aspect-ratio: 1.7777"
@media (aspect-ratio: (16/9)) {
h1{ color:green;}
}
// no evaluation: compiles into "min-width: 2+3"
@media (min-width: 2+3) {
h1{ color:green;}
}
// evaluation: compiles into "min-width: 5"
@media (min-width: (2+3)) {
h1{ color:green;}
}</pre><br />
<a name="ExpressionsandAmbiguityListsvsMathOperations"></a><h5>Lists vs Math Operations</h5>Css list members are separated either by comma or space. While comma causes no problems, space as list separator can lead to ambiguities. <br />
<br />
Consider following declaration: <pre class="brush:css">margin: 2 -1;</pre><br />
The expression <code>2 -1</code> could be interpreted either as subtraction with result <code>1</code> or as list with two members <code>2</code> and <code>-1</code>. If it is subtraction, then compiled style sheet should be equivalent to single margin-top declaration:<pre class="brush:css">margin-top: 1;</pre><br />
If it is list, then compiled style sheet should be equivalent to one margin-top and one margin-right declaration:<br />
<pre class="brush:css">margin-top: 2;
margin-right: -1;
</pre><br />
The problem: space inside an expression is ambiguous. All less.js versions solve this ambiguity in the same way. The only difference between them is planned for 2.0.0 and is caused by the expressions <a href="#BackwardIncompatibleChangesRequiredParentheses">must be enclosed in parentheses</a> backward incompatible change.<br />
<br />
<a name="ExpressionsandAmbiguityListsvsMathOperationsBreakdown"></a><h6>Breakdown</h6>First of all, there is no ambiguity in case of two numbers separated with two math operators:<ul><li>clear expression: <code>3 - -1</code>,</li>
<li>clear expression: <code>3- -1</code>,</li>
<li>clear expression: <code>3- +1</code>,</li>
<li>clear expression: <code>3-+1</code>.</li>
</ul><br />
There is no ambiguity in case of two numbers separated with operator other then plus/minus:<ul><li>clear expression: <code>3 *1</code>,</li>
<li>clear expression: <code>3*1</code>.</li>
</ul><br />
There is no ambiguity in case of two numbers separated by space only:<ul><li>clear list: <code>3 1</code>.</li>
</ul><br />
On the other hand, two numbers separated with exactly one plus or minus e.g., <code>1 -2</code>, are ambiguous. Their evaluation depends on exact spaces location. Spaces matter a lot if there is exactly one plus or minus between two numbers:<ul><li>ambiguous - space does matter: <code>3 -1</code>,</li>
<li>ambiguous - space does matter: <code>3 +1</code>,</li>
<li>ambiguous - space does matter: <code>3-1</code>,</li>
<li>ambiguous - space does matter: <code>3+ 1</code>,</li>
<li>ambiguous - space does matter: <code>3 + 1</code>.</li>
</ul><br />
<a name="ExpressionsandAmbiguityListsvsMathOperationsAmbiguitySolution"></a><h6>Ambiguity Solution</h6>If there is exactly one plus or minus between two numbers, less checks spaces. Only following sequence is evaluated as a list: <code>number</code>, <code>space</code>, <code>operator</code>, <code>number</code>. Anything else is treated as a math expression:<ul><li>expression: <code>3-1</code>,</li>
<li>expression: <code>3 - 1</code>,</li>
<li>expression: <code>3- 1</code>,</li>
<li>list: <code>3 -1</code>,</li>
<li>list: <code>3 +1</code>.</li>
</ul><br />
When compiler encounters complicated expression, it checks whether the expression is a list first. If it is a list, parser splits it into list members and evaluates each one separately. <br />
<br />
For example, the input <code>5 - 2 +3</code> is first split into a list with two members: <code>5 - 2</code> and <code>+3</code>. Each one is evaluated separately, so the result is another list with two members <code>3 +3</code>. <br />
<br />
Otherwise said, <code>5 - 2 +3</code> is effectively equivalent to <code>5 - 2, +3</code>.<br />
<br />
<a name="ExpressionsandAmbiguityListsvsMathOperationsLess1.3.xDetails"></a><h6>Less 1.3.x Details</h6>Old less.js versions make no difference between operations inside parentheses and outside of them. Without parentheses:<br />
<pre class="brush:css">#no-parentheses {
margin: 5 -2; // list: compiles into "margin: 5 -2;"
margin: 5 - 2; // expression: compiles into "margin: 3;"
margin: 5 - 2 +3; // expression in list: compiles into "margin: 3 +3;"
}
</pre><br />
Inside parentheses:<pre class="brush:css">#inside-parentheses {
margin: (5 -2); // list: compiles into "margin: 5 -2;"
margin: (5 - 2); // expression: compiles into "margin: 3;"
margin: (5 - 2 +3); // expression in list: compiles into "margin: 3 +3;"
margin: 5 - (2 +3); // list in expression: compiles into "margin: NaN;"
}
</pre><br />
Operations over expression inside parentheses:<br />
<pre class="brush:css">#multiply-parentheses {
margin: 2*(1 -2); // multiply list: compiles into "margin: NaN;"
margin: 2*(1 - 2); // multiply expression: compiles into "margin: -2;"
}
</pre><br />
<a name="ExpressionsandAmbiguityListsvsMathOperationsLess1.4.xDetails"></a><h6>Less 2.0.0 Details</h6>Less 2.0.0 will use exactly the same parse rules, the algorithm to distinguish between lists and expressions did not changed. All differences are caused by new <a href="#BackwardIncompatibleChangesRequiredParentheses">only expressions are allowed inside parentheses</a> rule.<br />
<br />
Content inside parentheses is evaluated as math. If it happens to be list, syntax error is thrown. The declaration that was accepted and compiled by 1.3.x version may suddenly start throwing errors:<br />
<pre class="brush:css">#inside-parentheses {
margin: (5 -2); // list in parentheses: syntax error
margin: (5 - 2); // expression: compiles into "margin: 3;"
margin: (5 - 2 +3); // list in parentheses: syntax error
margin: 5 - (2 +3); // list in parentheses: syntax error
}
</pre><br />
<a name="BackwardIncompatibleChanges"></a><h4>Backward Incompatible Changes</h4>Expression related backward incompatible changes are not the only planned changes, but they are definitely the most important ones. There are two planned changes and both have major consequences for existing less style sheets:<ul><li>less 2.0.0 <a href="#BackwardIncompatibleChangesRequiredParentheses">will evaluate only expressions enclosed inside parentheses</a>,</li>
<li>less 1.4.0 <a href="#BackwardIncompatibleChangesExpressionsandUnits">will change the way it treats operations over numbers with units</a>.</li>
</ul><br />
<a name="BackwardIncompatibleChangesRequiredParentheses"></a><h5>Required Parentheses</h5>Less 2.0.0 does not evaluate arithmetical operations unless they are be enclosed in parentheses. In addition, any expression enclosed in parentheses is expected to be math and less will attempt to evaluate it as a math.<br />
<br />
<a name="BackwardIncompatibleChangesRequiredParenthesesBasicExamples"></a><h6>Basic Examples</h6>If it is outside of parentheses, then it is not math and will not be evaluated:<pre class="brush:css">#no-parentheses {
margin: 1 - 2; // no evaluation: compiles into "margin: 1 - 2;"
margin: 1 -2; // no evaluation: compiles into "margin: 1 -2;"
margin: 2*(1 - 2); // no evaluation: compiles into "margin: 2*(1 - 2);"
}</pre><br />
Mathematical expressions inside parentheses are evaluated:<br />
<pre class="brush:css">#math-inside-parentheses {
margin: (1 - 2); // expression: compiles into "margin: -1;"
}</pre><br />
Non-mathematical expressions inside parentheses are evaluated and cause syntax errors: <br />
<pre class="brush:css">#math-inside-parentheses {
margin: (left); // identifier inside parentheses - syntax error
margin: (1 -2); // list inside parentheses - syntax error
}</pre><br />
<a name="BackwardIncompatibleChangesRequiredParenthesesMixins"></a><h6>Mixins</h6>Mixins parameters are treated the same way. The parameter is evaluated if and only if it is closed inside parentheses. It does not matter whether the parentheses is located inside the mixin declaration, inside the mixin call or somewhere else. If it is present somewhere, the expression is evaluated. Otherwise it is not.<br />
<br />
Less files with parentheses inside mixins body, inside mixin call and elsewhere:<br />
</div><table cellpadding="0"><tr><td><pre class="brush:css">/* Inside Body */
.mixin(@param) {
declaration: (@param);
}
#evaluated {
.mixin(3 - 1);
}
</pre></td><td><pre class="brush:css">/* Inside Call */
.mixin(@param) {
declaration: @param;
}
#evaluated {
.mixin((3 - 1));
}
</pre></td><td><pre class="brush:css">/* Elsewhere */
.mixin(@param) {
declaration: @param;
}
#evaluated {
@variable: (3 - 1);
.mixin(@variable);
}
</pre></td></tr>
</table><div style="text-align: justify;"><br />
Compilation result is the same in all three cases:<br />
<pre class="brush:css">/* Compiled css */
#evaluated {
declaration: 2;
}
</pre><br />
If the value inside parentheses happens to be list or other non-expression, syntax error is thrown. All three less files throw compilation error: <br />
</div><table cellpadding="0"><tr><td><pre class="brush:css">/* Inside Body */
.mixin(@param) {
declaration: (@param);
}
#evaluated {
/* !list! */
.mixin(3 -1);
}
</pre></td><td><pre class="brush:css">/* Inside Call */
.mixin(@param) {
declaration: @param;
}
#evaluated {
/* !list! */
.mixin((3 -1));
}
</pre></td><td><pre class="brush:css">/* Elsewhere */
.mixin(@param) {
declaration: @param;
}
#evaluated {
/* !list! */
@variable: (3 -1);
.mixin(@variable);
}
</pre></td></tr>
</table><div style="text-align: justify;"><br />
If there is no parentheses, expression will not be evaluated. Both input files compile into the same css:<br />
<div><table cellpadding="0"><tr><td><pre class="brush:css">/* Less input 1 */
.mixin(@param) {
declaration: @param;
}
#not-evaluated {
.mixin(3 - 1);
}
</pre></td> <td><pre class="brush:css">/* Less input 2 */
.mixin(@param) {
declaration: @param;
}
#not-evaluated {
@variable: 3 - 1;
.mixin(@variable);
}
</pre></td> <td><pre class="brush:css">/* Compiled css */
#not-evaluated {
declaration: 3 - 1;
}
</pre></td></tr>
</table></div><br />
<a name="BackwardIncompatibleChangesRequiredParenthesesVariables"></a><h6>Variables</h6>Variables are treated with exactly the same logic. Three less files with parentheses on three different places:<br />
</div><table cellpadding="0"><tr><td><pre class="brush:css">/* Less 1 */
#evaluated {
@var1: (3 - 1);
@var2: @var1;
declaration: @var2;
}</pre></td><td><pre class="brush:css">/* Less 2 */
#evaluated {
@var1: 3 - 1;
@var2: (@var1);
declaration: @var2;
}</pre></td><td><pre class="brush:css">/* Less 3 */
#evaluated {
@var1: 3 - 1;
@var2: @var1;
declaration: (@var2);
}</pre></td></tr>
</table><div style="text-align: justify;"><br />
Compilation result is the same in all three cases:<br />
<pre class="brush:css">/* Compiled css */
#evaluated {
declaration: 2;
}
</pre><br />
No parentheses - no evaluation:<br />
<div><table cellpadding="0"><tr><td><pre class="brush:css">/* Less input */
#not-evaluated {
@var1: 3 - 1;
@var2: @var1;
declaration: @var2;
}</pre></td> <td><pre class="brush:css">/* Compiled css */
#not-evaluated {
declaration: 3 - 1;
}
</pre></td></tr>
</table></div><br />
<a name="BackwardIncompatibleChangesRequiredParenthesesFunctions"></a><h6>Functions</h6>Built-in functions are evaluated even without parentheses around them. However, if you want to use an expression as a function parameter, the expression must be enclosed in parentheses.<br />
<br />
Expression as a function parameter and compilation result:<br />
<div><table cellpadding="0"><tr><td><pre class="brush:css">/* Less input */
#function {
declaration: sqrt((36-11));
}</pre></td> <td><pre class="brush:css">/* Compiled css */
#function {
declaration: 5;
}
</pre></td></tr>
</table></div><br />
Expression without parentheses causes syntax errors:<br />
<pre class="brush:css">/* Syntax error */
#function {
declaration: sqrt(36-11); // syntax error
}</pre><br />
<a name="BackwardIncompatibleChangesExpressionsandUnits"></a><h5>Expressions and Units</h5>The old way of treating units inside expressions is very simple. Less 1.3.x takes numbers as they are, calculates the result and then assigns leftmost explicitly stated unit type to the result.<br />
<br />
The old way:<br />
<pre class="brush:css">#selector {
property: 2 + 5cm - 3mm; // compiles into "property: 4cm"
property: (1 + 5cm) - 3%; // compiles into "property: 3cm"
}</pre><br />
Less 1.4.0 treats units in a bit more interesting way. While division and multiplication have been left intact, plus and minus operations gained new ability:<ul><li>plus <code>+</code> and minus <code>-</code> operations convert between compatible number types.</li>
</ul><br />
<a name="BackwardIncompatibleChangesExpressionsandUnitsUnitConversionsforPlusandMinus"></a><h6>Unit Conversions for Plus and Minus</h6>Plus <code>+</code> and minus <code>-</code> operations convert between compatible number types. Numbers without units are not converted:<br />
<pre class="brush:css">#selector-compiles {
/* less converts 3mm into 0.3cm before the operation */
property: (5cm - 3mm); // compiles into "property: 4.7cm;"
property: (2 + 5cm - 3mm); // compiles into "property: 6.7cm;"
}
</pre><br />
If you add or subtract incompatible numbers, the result inherits left side unit. Right side unit is ignored and behaves as if it would be unspecified:<br />
<pre class="brush:css">#selector-wrong {
/* % is incompatible with cm and ignored */
property-zero: (5cm - 3%); // compiles into "2cm;"
/* % is incompatible with cm and ignored */
property-one: (5cm - 3% - 1mm); // compiles into "1.9cm;"
/* % is incompatible with cm and ignored */
property-two: ((1 + 5cm) - 3% - 1mm); // compiles into "2.9cm;"
}</pre><br />
<a name="BackwardIncompatibleChangesExpressionsandUnitsUnitsExpressionsSimplificationforDivisionandMultiplication"></a><h6>Notes on Division and Multiplication</h6>Unlike plus and minus, division and multiplication are not able to convert between unit types. Just as in the old version, they take numbers as they are, calculate the result and then assign leftmost explicitly stated unit type to the result:<br />
<pre class="brush:css">#simplified-expressions {
property-one: (4cm/2cm); // css: "property-one: 2cm;"
property-two: (4cm*2mm); // css: "property-two: 8cm;"
property-three: (1/1mm); // css: "property-three: 1mm;"
</pre><br />
Plus and minus over divided or multiplicated expressions still convert units:<br />
<pre class="brush:css">#syntax-errors {
property-1: (2cm/1mm - 1mm/1cm); // css: "1.9cm"
property-2: (2cm*1mm + 1mm*1cm); // css: "2.1cm"
}
</pre><br />
<a name="Preview"></a><h4>Preview Availability</h4>If you want to try the new syntax, less.js already released fourth beta of upcoming 1.4.0. It is not meant to be used in production, but you can try it out anyway or maybe even start to migrate old style sheets. And if you are writing new style sheet, it is probably good idea to make it new syntax compatible from the start. <br />
<br />
However, fourth beta does not require parentheses around math expressions. That change was included in previous beta three and then <a href="https://github.com/cloudhead/less.js/issues/1259">postponed</a>. So, if you want to play around required parentheses, you have three options:<ul><li>get it from npm: <code>npm install -g less@1.4.0-b3</code></li>
<li>get it from <a href="https://github.com/cloudhead/less.js/commits/master/dist/less-1.4.0-beta.js">history on Github</a>,</li>
<li>wait for first 2.0.0 beta release.</li>
</ul><br />
<a name="Conclusion"></a><h4>Conclusion</h4>That would be all about expressions, quirks and their incoming changes. Outside of what was described in this post, expressions in less work the same way as expressions in any other language.<br />
<br />
In short, all you need to remember is this:<ul><li>Place all expressions into parentheses.</li>
<li>If expressions return unexpected results, check out units.</li>
<li>If it throws unexpected errors or generates lists where it should not, check out spaces in expressions.</li>
</ul></div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com3tag:blogger.com,1999:blog-3355333693246614846.post-72246909463141095742013-03-31T01:31:00.000-07:002013-05-09T03:31:55.723-07:00Less CSS - Tips and Trics<style>div.blogger-clickTrap {display: none;}</style><style type="text/css">div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}</style><div style="text-align: justify;">Less is declarative language designed to behave and look as closely to css as possible. Less started as an attempt to make css style sheets easier to read and maintain and then added more features and abilities. It evolved so much that it became theoretically possible to do anything with it. <br />
<br />
This post shows some less known but very useful less features. It also contains list of potentially useful workarounds and tricks you can use if the default less syntax is not sufficient for what you need. Finally, last chapter shows one practical application of those tricks. We will create mixin to auto generate vendor prefixes for css declarations.<a name='more'></a><br />
<br />
<a name="TableOfContents"></a><h4>Table Of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<ul><li><a href="#Basics">Basics</a></li>
<li><a href="#Compilers">Compilers</a></li>
<li><a href="#LessKnownLessAbilities">Less Known Less Abilities</a></li>
<ul><li><a href="#LessKnownLessAbilitiesMixinsWithReturnValues">Mixins With Return Values</a></li>
<li><a href="#LessKnownLessAbilitiesSemicolonasMixinArgumentsSeparator">Semicolon as Mixin Arguments Separator</a></li>
<li><a href="#LessKnownLessAbilitiesMediaQueryinVariable">Media Query in Variable</a></li>
<li><a href="#LessKnownLessAbilitiesImport">Import</a></li>
</ul><li><a href="#TricksandWorkarounds">Tricks and Workarounds</a></li>
<ul><li><a href="#TricksandWorkaroundsTripleVariableIndirection">Triple Variable Indirection</a></li>
<ul><li><a href="#TricksandWorkaroundsTripleVariableIndirectionWhatWorks">What Works</a></li>
<li><a href="#TricksandWorkaroundsTripleVariableIndirectionWhatDoesNotWork">What Does Not Work</a></li>
<li><a href="#TricksandWorkaroundsTripleVariableIndirectionWorkaround">Workaround</a></li>
</ul><li><a href="#TricksandWorkaroundsInterpolatePropertyNames">Interpolate Property Names</a></li>
<ul><li><a href="#TricksandWorkaroundsInterpolatePropertyNamesWhatWorks">What Works</a></li>
<li><a href="#TricksandWorkaroundsInterpolatePropertyNamesWhatDoesNotWork">What Does Not Work</a></li>
<li><a href="#TricksandWorkaroundsInterpolatePropertyNamesWorkaround">Workaround</a></li>
</ul><li><a href="#TricksandWorkaroundsLooping">Looping</a></li>
<ul><li><a href="#TricksandWorkaroundsLoopingWhatDoesNotWork">What Does Not Work</a></li>
<li><a href="#TricksandWorkaroundsLoopingWorkaround">Workaround</a></li>
<li><a href="#TricksandWorkaroundsLoopingWhoWouldUseThat">Who Would Use That</a></li>
<li><a href="#TricksandWorkaroundsLoopingExample:LoopThroughList">Example: Loop Through List</a></li>
</ul></ul><li><a href="#GenerateVendorPrefixes">Generate Vendor Prefixes</a></li>
<ul><li><a href="#GenerateVendorPrefixesAPI">API</a></li>
<li><a href="#GenerateVendorPrefixesHowItWorks">How It Works</a></li>
<li><a href="#GenerateVendorPrefixesFullCode">Full Code</a></li>
</ul><li><a href="#ToBeContinued">To Be Continued</a></li>
</ul><br />
</div></div><a name="Basics"></a><h4>Basics</h4>We assume that you know both what less is and its basic features. If you know how less variables, mixins and nesting work, then you know enough to read this post. If you never used them, you can start either with <a href="http://lesscss.org/">official documentation</a> that belongs to original JavaScript compiler less.js or <a href="https://github.com/SomMeri/less4j/wiki/Supported-Less-Language">detailed language description</a> written as part of java compiler less4j. <br />
<br />
You can also read an introductory article about less written on this <a href="http://meri-stuff.blogspot.sk/2012/06/inheritance-and-variables-in-css-with.html">blog</a>.<br />
<br />
<a name="Compilers"></a><h4>Compilers</h4>Just few weeks ago, we released first non beta version of less compiler java port called <a href="https://github.com/SomMeri/less4j#readme">less4j</a>. Feel free to try it out.<br />
<br />
Canonical compiler, the one whose behavior defines the language, is written in JavaScript and called <a href="http://lesscss.org/">less.js</a>. It works both in browser and in node.js environment. We tried it with windows node.js and npm. The combination runs without hustle or problems. <br />
<br />
Next less.js version 1.4.0 will be backward incompatible, but for good reasons. Changes are meant to solve some old quirks and will be partially described in our next post. Of course, less4j will follow less.js changes as soon as possible.<br />
<br />
Lists of <a href="https://github.com/cloudhead/less.js/wiki/Ports-of-LESS">other ports</a> and <a href="https://github.com/cloudhead/less.js/wiki/Editors-and-Plugins">GUI editors</a> are maintained on less.js wiki page.<br />
<br />
<a name="LessKnownLessAbilities"></a><h4>Less Known Less Abilities</h4>This chapter contains four less features. Three of them have been added only recently and one was documented only in less.js issue database. <br />
<br />
The list of features covered by this chapter: <ul><li><a href="#LessKnownLessAbilitiesMixinsWithReturnValues">mixins with return values</a>,</li>
<li><a href="#LessKnownLessAbilitiesSemicolonasMixinArgumentsSeparator">semicolon as mixin arguments separator</a>,</li>
<li><a href="#LessKnownLessAbilitiesMediaQueryinVariable">media query in variable</a>,</li>
<li><a href="#LessKnownLessAbilitiesImport">import into ruleset or mixin</a>.</li>
</ul><br />
<a name="LessKnownLessAbilitiesMixinsWithReturnValues"></a><h5>Mixins With Return Values</h5>Variables declared in mixin can act as return values. This is important not only because it can be useful occasionally, but also because this feature has one rather important gotcha. <br />
<br />
Variables act as mixins return values:<br />
<pre class="brush:css">.mixin() { @unique: 45; } /* mixin defines variable */
#call-mixin {
variable-value: @unique; /* print variable defined inside mixin */
.mixin(); /* call the mixin */
}
</pre><br />
All variables defined in mixin are copied into callers scope and can rewrite local variables. In addition, any mixin can call other mixins and those other mixins can call yet another mixins. All that can create an arbitrary long chain of mixins calls. Any of them may modify value of any variable defined in the original caller scope.<br />
<br />
The dangerous part "can rewrite local variables" will be fixed in the next 1.4.0 version. Next less.js release will copy only those variables, that can not possibly rewrite local values. <br />
<br />
Consider following less:<br />
<pre class="brush:css">.mixin() { @clash: 45; @unique: 45; } /* mixin defines two variables */
#call-mixin {
@clash: 0; /* variable with the same name as the one inside .mixin */
/* print actual variables values */
unique: @unique;
clash: @clash;
.mixin(); /* call the mixin */
}
</pre><br />
compiled with less 1.3.3:<br />
<pre class="brush:css">#call-mixin {
unique: 45; /* variable defined inside mixin was accessible */
clash: 45; /* variable defined inside mixin rewrote local value */
}</pre><br />
compiled with less 1.4.0:<br />
<pre class="brush:css">#call-mixin {
unique: 45; /* variable defined inside mixin was accessible */
clash: 0; /* local variable kept its value*/
}</pre><br />
Note: less 1.4.0 will protect only local variables. Variables found in any other scope, for example global scope, are not protected.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
Non local variable is not protected: <br />
<pre class="brush:css">/* mixin with return value */
.mixin() { @clash: 45; }
/* variable with the same name .mixin return value */
@clash: 0;
#call-mixin {
clash: @clash; /* print variables value */
.mixin(); /* call the mixin */
}
</pre><br />
less 1.4.0 compilation result:<br />
<pre class="brush:java">#call-mixin {
/* variable defined outside of local scope was not protected */
clash: 45;
}
</pre><br />
</div></div><a name="LessKnownLessAbilitiesSemicolonasMixinArgumentsSeparator"></a><h5>Semicolon as Mixin Arguments Separator</h5>Less.js 1.3.3 added semicolon as mixin arguments separator. Both <code>.call(1; 2; 3)</code> and <code>.call(1, 2, 3)</code> are now valid mixin calls and do the same thing. Mixin call with semicolon separated arguments may look weird and unusual, but there was good reason for this addition.<br />
<br />
The symbol comma has meaning in standard css. It is list separator and <code>1, 2, 3</code> is valid list containing three numbers. Comma acting as mixin arguments separator made it impossible to use comma separated lists as arguments. <br />
<br />
If the latest less compiler sees at least one semicolon inside mixin call or declaration, then it assumes that arguments are separated by semicolons and all commas belong to css lists:<br />
<pre class="brush:css">/* two arguments, both with comma separated list */
.call(1, 2, 3; something, else);
/* three arguments and each contains one number */
.call(1, 2, 3);
/* one argument containing comma separated css list */
.call(1, 2, 3;); // note dummy semicolon
/*comma separated default value */
.declaration(@param1: red, blue;) {
// mixin content
}</pre><br />
<a name="LessKnownLessAbilitiesMediaQueryinVariable"></a><h5>Media Query in Variable</h5>Last releases of less have ability to store media queries in variables and use those variables inside <code>@media</code> rules. Use it whenever you need the same media query at multiple places. If you decide to modify that media query, you will have only one place to change.<br />
<br />
Store and use <code>tablet and (min-width: 500px)</code> media query:<br />
<pre class="brush:css">/* note '~' in the beginning - media query must be escaped */
@singleQuery: ~"tablet and (min-width: 500px)";
@media screen, @singleQuery {
set {
padding: 3 3 3 3;
}
}</pre><br />
compiles into:<br />
<pre class="brush:css">@media screen, tablet and (min-width: 500px) {
set {
padding: 3 3 3 3;
}
}</pre><br />
<a name="LessKnownLessAbilitiesImport"></a><h5>Import</h5>Less import statement is not limited to style sheet top level. It is possible to import less files into rulesets and mixins. The feature helps to deal with name conflicts between multiple less libraries or less libraries versions. <br />
<br />
<pre class="brush:css">.namespace-1-2-3() {
@import "library-1.2.3.less";
}
.namespace-2-0-0() {
@import "library-2.0.0.less";
}
#use-both-libraries-without-name-conflicts {
.namespace-1-2-3 > .usefull-mixin();
.namespace-2-0-0 > .usefull-mixin();
}
</pre><br />
<a name="TricksandWorkarounds"></a><h4>Tricks and Workarounds</h4>Some occasionally demanded but not yet directly supported features are achievable if you are willing to use a workaround. This chapter shows three such tricks: <ul><li><a href="#TricksandWorkaroundsTripleVariableIndirection">triple variable indirection</a>,</li>
<li><a href="#TricksandWorkaroundsInterpolatePropertyNames">interpolate property names</a>,</li>
<li><a href="#TricksandWorkaroundsLooping">create a loop</a>.</li>
</ul><br />
<a name="TricksandWorkaroundsTripleVariableIndirection"></a><h5>Triple Variable Indirection</h5>Related less.js issue: <a href="https://github.com/cloudhead/less.js/issues/1188">Variable Name Interpolation</a><br />
<br />
Used less features:<ul><li>variables,</li>
<li>variable references,</li>
<li>string interpolation.</li>
</ul><br />
<a name="TricksandWorkaroundsTripleVariableIndirectionWhatWorks"></a><h6>What Works</h6>Less supports variables and any variable holding string can act as reference to another variable. If you want to use references, you have to put additional <code>@</code> before the variable name e.g., <code>@@reference</code>.<br />
<br />
Use the <code>@reference</code> variable as a reference to the <code>@number</code> variable:<br />
<pre class="brush:css">@number: 10;
@reference: "number";
h1 { size: @@reference; }
</pre><br />
output:<br />
<pre class="brush:css">h1 { size: 10; }
</pre><br />
<a name="TricksandWorkaroundsTripleVariableIndirectionWhatDoesNotWork"></a><h6>What Does Not Work</h6>Less does not support triple variable indirection. This <code>@@@referenceToReference</code> is not valid less:<br />
<pre class="brush:css">@number: 10;
@reference: "number";
@referenceToReference: "reference"; // desperate attempt
h1 { size: @@@referenceToReference; } //syntax error
</pre><br />
<a name="TricksandWorkaroundsTripleVariableIndirectionWorkaround"></a><h6>Workaround</h6>Use string interpolation to compose referenced variable name:<br />
<pre class="brush:css">@number: 10;
@reference: "number";
/* compose indirect variable name inside another variable */
@referenceToReference: "@{reference}";
h1 { size: @@referenceToReference; }
</pre><br />
previous less compiles into following css:<br />
<pre class="brush:css">h1 { size: 10; }</pre><br />
Bonus: string interpolation allows you to go arbitrary far:<br />
<pre class="brush:css">@number: 10;
@reference: "number";
@referenceToReference: "@{reference}";
/* chaining of interpolated variables can be arbitrary long */
@referenceToReference2: "@{referenceToReference}";
@referenceToReference3: "@{referenceToReference2}";
@referenceToReference4: "@{referenceToReference3}";
@referenceToReference5: "@{referenceToReference4}";
h1 { size: @@referenceToReference5; }
</pre><br />
<a name="TricksandWorkaroundsInterpolatePropertyNames"></a><h5>Interpolate Property Names</h5>Related less.js issue: <a href="https://github.com/cloudhead/less.js/issues/1199">Auto Prefix replacement</a><br />
<br />
Used less features:<ul><li>escaped values,</li>
<li>variables.</li>
</ul><br />
<a name="TricksandWorkaroundsInterpolatePropertyNamesWhatWorks"></a><h6>What Works</h6>Any content can be stored into variable and then used<ul><li>as property value,</li>
<li>inside string or escaped value,</li>
<li>as selector,</li>
<li>as media query.</li>
</ul><br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
Sample input:<br />
<pre class="brush:java">@property_value: 1px;
@selector: ~"div.someclass";
@media_query: ~"tablet";
@media @media_query {
@{selector} {
padding: @property_value;
in-string: "Padding was set to: @{property_value}";
}
}
</pre><br />
compiles into:<br />
<pre class="brush:java">@media tablet {
div.someclass {
padding: 1px;
in-string: "Padding was set to: 1px";
}
}
</pre><br />
</div></div><a name="TricksandWorkaroundsInterpolatePropertyNamesWhatDoesNotWork"></a><h6>What Does Not Work</h6>Property names can not be rendered dynamically. You can not use variables or escaped values to generate property names:<br />
<pre class="brush:css">@css-property-name: "size";
#desperate-1 {
@css-property-name: 10; // syntax error
}
#desperate-2 {
@{css-property-name}: 10; // syntax error
}
</pre><br />
<a name="TricksandWorkaroundsInterpolatePropertyNamesWorkaround"></a><h6>Workaround</h6>Browsers ignore declarations with an unknown property. The declaration <code>hack: 1;</code> will be ignored by all browsers, so we do not have to care whether generated css contains it or not. <br />
<br />
In addition, less escaping allows you to place any content directly into generated css. Escaped values use <code>~"escaped value"</code> syntax and the intention was to allow browser hacks, invalid css or proprietary syntax otherwise unrecognized by Less. Regardless of original intentions, this <code>hack: 1 ~"; something"</code> will be rendered as <code>hack: 1 ; something</code>, so we can abuse it.<br />
<br />
Render property name from a variable:<br />
<pre class="brush:css">@css-property-name: "size";
#workaround {
hack: 1 ~"; @{css-property-name}:" 10;
}</pre><br />
previous less compiles into following css:<br />
<pre class="brush:css">#workaround {
hack: 1 ; size: 10;
}</pre><br />
<a name="TricksandWorkaroundsLooping"></a><h5>Looping</h5>Related less.js issue: <a href="https://github.com/cloudhead/less.js/issues/869">Idea: loop support</a><br />
<br />
Used less features:<ul><li>mixins,</li>
<li>guards,</li>
<li>expressions.</li>
</ul><br />
<a name="TricksandWorkaroundsLoopingWhatDoesNotWork"></a><h6>What Does Not Work</h6>Less is declarative language and does not support looping. It does not have native <code>while</code>, <code>for</code>, <code>repeat ... until</code> or any other cycling structure.<br />
<br />
<a name="TricksandWorkaroundsLoopingWorkaround"></a><h6>Workaround</h6>Use recursive mixins calls to simulate the loop. The most simple looping mixin needs only one parameter - how many times should it run the loop. It does whatever it needs to do and then calls itself with smaller parameter. Looping stops once the parameter reaches zero.<br />
<br />
Any cycling structure can be simulated this or similar way, but you have to be careful. It is very easy to write forever running mixin.<br />
<br />
Create looping mixin:<br />
<pre class="brush:css">/* looping mixin */
.loop (@index) when (@index > 0) {
loop: member @index; /* do whatever you need to do */
.loop((@index - 1)); /* loop again */
}
/* stop mixin */
.loop (@index) when (@index <= 0) {
loop: end;
}
/* use looping mixin */
#run-loop-5-times {
.loop(5);
}</pre><br />
compiles into:<br />
<pre class="brush:css">#run-loop-5-times {
loop: member 5;
loop: member 4;
loop: member 3;
loop: member 2;
loop: member 1;
loop: end;
}</pre><br />
Note: we used double parentheses in <code>.loop((@index - 1))</code> call. This was intentional and not a mistake. Future less 1.4.0 will require parentheses around math operation, so we added them to have future-proof code example.<br />
<br />
<a name="TricksandWorkaroundsLoopingWhoWouldUseThat"></a><h6>Who Would Use That</h6><a href="http://twitter.github.com/bootstrap/">Twitter Bootstrap</a> uses looping to generate <a href="http://twitter.github.com/bootstrap/scaffolding.html#gridSystem">the grid</a>: <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:css">/* Simplified Twitter Bootstrap Code */
.core (@gridColumns, @gridColumnWidth) {
.spanX (@index) when (@index > 0) { /* looping mixin */
(~".span@{index}") { .span(@index); } /* generate column */
.spanX(@index - 1); /* loop again */
}
.spanX (0) {} /* stop mixin */
.span (@columnIndx) {
width: (@gridColumnWidth * @columnIndx);
}
// generate .spanX and .offsetX
.spanX (@gridColumns);
}
/* generate the grid */
.sample-grid {
@gridColumns: 3;
@gridColumnWidth: 55;
.core(@gridColumns, @gridColumnWidth);
}</pre><br />
compiles into:<br />
<pre class="brush:css">.sample-grid .span3 {
width: 165;
}
.sample-grid .span2 {
width: 110;
}
.sample-grid .span1 {
width: 55;
}
</pre><br />
</div></div><a name="TricksandWorkaroundsLoopingExample:LoopThroughList"></a><h6>Example: Loop Through List</h6>Mixin to loop through list of strings or escaped values does not end when the <code>index</code> reaches zero, it checks type of the next list member instead:<br />
<pre class="brush:css">.loop-strings(@list, @index: 1) when (isstring(extract(@list, @index))) {
@currentMember: extract(@list, @index);
do-something-with: @currentMember;
.loop-strings(@list, (@index + 1)); /* loop the next member */
}
#use-place {
// loop through list of escaped values
.loop-strings(~"A", ~"B", ~"C", ~"1", ~"2", ~"3";);
}</pre><br />
compiles into:<br />
<pre class="brush:css">#use-place {
do-something-with: A;
do-something-with: B;
do-something-with: C;
do-something-with: 1;
do-something-with: 2;
do-something-with: 3;
}</pre><br />
<a name="GenerateVendorPrefixes"></a><h4>Generate Vendor Prefixes</h4>Auto generation of vendor prefixes in css declarations is one of those things that are repeatedly asked for in less.js issues database. While less currently does not have such feature, it is possible to combine above workarounds into mixin that auto-generates them. <br />
<br />
<a name="GenerateVendorPrefixesAPI"></a><h5>API</h5>The mixin <code>.vendorPrefixes(@prefixes, @declaration, @values...)</code> has two mandatory arguments followed by any number of optional arguments:<ul><li><code>@prefixes</code> - mandatory list of prefixes,</li>
<li><code>@declaration</code> - mandatory string with the declared property,</li>
<li><code>@values...</code> - optional - all declared values.</li>
</ul><br />
This solution is not perfect, the declaration to be rendered has to be written inside string and sent as mixin parameters: <pre class="brush:css">.vendorPrefixes(@list-of-prefixes; "transition"; all 4s ease);</pre><br />
That call has certainly lower then ideal readability. On the other hand, this solution shows full power of less and combines two different tricks into one solution, so it still can be useful for demonstration purposes.<br />
<br />
<a name="GenerateVendorPrefixesHowItWorks"></a><h5>How It Works</h5>Rendering mixin uses <a href="#TricksandWorkaroundsInterpolatePropertyNames">interpolate property names</a> trick to render vendor prefixed declaration:<br />
<pre class="brush:css">/* Render exactly vendor prefixed declaration. */
.renderWithPrefix(@prefix; @declaration; @values...) {
hack: 1 ~"; @{prefix}-@{declaration}: " @values;
}
</pre><br />
<a href="#TricksandWorkaroundsLooping">Looping</a> mixin goes through all prefixes:<br />
<pre class="brush:css">.nthVendorPrefix(@prefixes; @index; @declaration; @values...)
when (isstring(extract(@prefixes, @index))) {
@prefix: extract(@prefixes, @index);
.renderWithPrefix(@prefix, @declaration, @values);
.nthVendorPrefix(@prefixes; (@index + 1); @declaration; @values);
}</pre><br />
Finally, "API" mixin to starts the looping-rendering process:<br />
<pre class="brush:css">.vendorPrefixes(@prefixes, @declaration, @values...) {
.nthVendorPrefix(@prefixes; 1; @declaration; @values);
}</pre><br />
<a name="GenerateVendorPrefixesFullCode"></a><h5>Full Code</h5><pre class="brush:css">/* Render exactly vendor prefixed declaration. */
.renderWithPrefix(@prefix; @declaration; @values...) {
hack: 1 ~"; @{prefix}-@{declaration}: " @values;
}
/* Looping mixin assumes that @prefixes list contains strings:
* finds @index-th prefix in @prefixes list,
* renders one vendor prefixed declaration,
* loop: call itself with higher @index.
Guard prevents indefinite looping, it will stop once @index goes off the list.
*/
.nthVendorPrefix(@prefixes; @index; @declaration; @values...)
when (isstring(extract(@prefixes, @index))) {
@prefix: extract(@prefixes, @index);
.renderWithPrefix(@prefix, @declaration, @values);
.nthVendorPrefix(@prefixes; (@index + 1); @declaration; @values);
}
/* Render exactly vendor prefixed declaration. */
.vendorPrefixes(@prefixes, @declaration, @values...) {
.nthVendorPrefix(@prefixes; 1; @declaration; @values);
}
#use-place {
@list-of-prefixes: ~"-moz", ~"-ie", ~"-webkit";
.vendorPrefixes(@list-of-prefixes; "transition"; all 4s ease);
}</pre><br />
compiles into:<pre class="brush:css">#use-place {
hack: 1 ; -moz-transition: all 4s ease;
hack: 1 ; -ie-transition: all 4s ease;
hack: 1 ; -webkit-transition: all 4s ease;
}</pre><br />
<a name="ToBeContinued"></a><h4>To Be Continued</h4><a href="http://meri-stuff.blogspot.sk/2013/05/less-css-expressions-and-traps.html">Next post</a> is about less expressions and their planned backward incompatible changes. <br />
<br />
In most cases, expressions are easy to use and very useful. Unfortunately, their interactions with the rest of the css syntax can occasionally cause somewhat quirky behavior. <a href="http://meri-stuff.blogspot.sk/2013/05/less-css-expressions-and-traps.html">Next post</a> shows when it happens, what will be solved by planned changes and which traps will still remain there.<br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>
Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com9tag:blogger.com,1999:blog-3355333693246614846.post-82584817921627954382012-12-17T00:55:00.000-08:002012-12-17T00:55:10.105-08:00ANTLR - Semantic Predicates<style>div.blogger-clickTrap {display: none;}</style><style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Parsing simple grammar with <a href="http://www.antlr.org/">antlr</a> is <a href="http://meri-stuff.blogspot.sk/2011/08/antlr-tutorial-hello-word.html">simple</a>. All you have to do is to use regular expressions to <a href="http://meri-stuff.blogspot.sk/2011/09/antlr-tutorial-expression-language.html">describe your language</a> and let antlr generate lexer and parser. Parsing big or complicated languages occasionally require more, because describing them in regular expressions only can be hard or even impossible.<br />
<br />
Semantic predicates are java (or C++, or JavaScript, or ...) conditions written inside the grammar. Antlr uses them either to choose between multiple alternatives or as additional assertions to be checked. They can be placed inside both lexer and parser, but this post focus only on their usage within the parser. They add a lot of power to antlr.<a name='more'></a><br />
<br />
This post assumes that you have general idea on <a href="http://meri-stuff.blogspot.sk/2011/08/antlr-tutorial-hello-word.html">what is antlr</a> and how generated parser <a href="http://meri-stuff.blogspot.sk/2011/09/antlr-tutorial-expression-language.html#ParserBasics">works</a>. If you do not, please read linked posts first. They contain everything needed.<br />
<br />
First chapter contains two motivational use cases. Second chapter describes syntax, terminology and shows simple failed semantic predicate. The post then explains how semantic predicates influence prediction and generated code. It also shows how to write useful conditions and how to solve initial use cases. Final chapter wraps it all into short conclusion.<br />
<br />
All <a href="https://github.com/SomMeri/org.meri.antlr.predicates/tree/master/src/main/antlr3/org/meri/antlr/predicates">examples and grammars</a> used in this post are available <a href="https://github.com/SomMeri/org.meri.antlr.predicates">on Github</a>.<br />
<br />
<a name="TableofContents"></a><h4>Table of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><ul><li><a href="#UseCases">Use Cases</a></li>
<ul><li><a href="#UseCasesKeywordsnth">Keywords - nth</a></li>
<li><a href="#UseCasesSignificantWhitespaces">Significant Whitespaces</a></li>
</ul><li><a href="#Basics">Basics</a></li>
<ul><li><a href="#BasicsSyntax">Syntax</a></li>
<li><a href="#BasicsTerminology">Terminology</a></li>
<ul><li><a href="#BasicsTerminologyDisambiguatingSemanticPredicate">Disambiguating Semantic Predicate</a></li>
<li><a href="#BasicsTerminologyValidatingSemanticPredicate">Validating Semantic Predicate</a></li>
<li><a href="#BasicsTerminologyGatedSemanticPredicate">Gated Semantic Predicate</a></li>
</ul><li><a href="#BasicsFailedPredicates">Failed Predicates</a></li>
</ul><li><a href="#HoistingandPrediction">Hoisting and Prediction</a></li>
<ul><li><a href="#HoistingandPredictionWhatItIs">What It Is</a></li>
<li><a href="#HoistingandPredictionConsequences">Consequences</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsed">When It Is Used</a></li>
<ul><li><a href="#HoistingandPredictionWhenItIsUsedIfNeededHoistingDisambiguatingPredicate">If Needed Hoisting - Disambiguating Predicate</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsedAlwaysHoistingGatedRules">Always Hoisting - Gated Rules</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsedNeverHoistingMiddleofaRule">Never Hoisting - Middle of a Rule</a></li>
</ul><li><a href="#HoistingandPredictionWhenItIsUsedNuances">Nuances</a></li>
<ul><li><a href="#HoistingandPredictionWhenItIsUsedNuancesDisambiguatingPredicatesAdvanced">Disambiguating Predicates - Advanced</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsedNuancesLoops">Loops</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsedNuancesUncoveredAlternatives">Uncovered Alternatives</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsedNuancesCombinationofGatedandDisambiguatedPredicates">Combination of Gated and Disambiguated Predicates</a></li>
<li><a href="#HoistingandPredictionWhenItIsUsedNuancesAdditionalParenthesis">Additional Parenthesis</a></li>
</ul></ul><li><a href="#HoistingandPredictionWhenItIsUsedNuancesBacktracking">Backtracking</a></li>
<li><a href="#WritingConditions">Writing Conditions</a></li>
<ul><li><a href="#WritingConditionsInputTokenStream">Input Token Stream</a></li>
<ul><li><a href="#WritingConditionsInputTokenStreamExamples">Examples</a></li>
</ul><li><a href="#WritingConditionsLabelsandLists">Labels and Lists</a></li>
<ul><li><a href="#WritingConditionsLabelsandListsLabelExample">Label Example</a></li>
<li><a href="#WritingConditionsLabelsandListsLabelListExample">Label List Example</a></li>
<li><a href="#WritingConditionsLabelsandListsUndefinedLabels">Undefined Labels</a></li>
<li><a href="#WritingConditionsLabelsandListsLabelsandHoisting">Labels and Hoisting</a></li>
</ul><li><a href="#WritingConditionsAccesstoLocalVariables">Access to Local Variables</a></li>
</ul><li><a href="#SolvingInitialUseCases">Solving Initial Use Cases</a></li>
<ul><li><a href="#SolvingInitialUseCasesKeywordsnth">Keywords - nth</a></li>
<li><a href="#SolvingInitialUseCasesSignificantWhitespaces">Significant Whitespaces</a></li>
</ul><li><a href="#WrappingItUp">Wrapping It Up</a></li>
<ul><li><a href="#WrappingItUpValidatingSemanticPredicates">Validating Semantic Predicates</a></li>
<li><a href="#WrappingItUpDisambiguatingSemanticPredicates">Disambiguating Semantic Predicates</a></li>
<li><a href="#WrappingItUpGatedSemanticPredicates">Gated Semantic Predicates</a></li>
</ul><li><a href="#Resources">Resources</a></li>
</ul></div></div><a name="UseCases"></a><h4>Use Cases</h4>As we spend some time <a href="http://meri-stuff.blogspot.sk/2012/11/project-report-less4j.html">parsing css-like language</a>, both our use cases describe problem we had to solve while writing css part of that parser. First one is about issue encountered while working on pseudo classes and second deals with tricky whitespaces in selectors.<br />
<br />
If you never worked with css or are not interested in use cases, <a href="#Basics">skip</a> this chapter.<br />
<br />
<a name="UseCasesKeywordsnth"></a><h5>Keywords - nth</h5>Some css <a href="http://www.w3.org/TR/selectors/#pseudo-classes">pseudo</a> <a href="http://css-tricks.com/pseudo-class-selectors/">classes</a> require parameter which can be either a number, an identifier, a selector or something called nth expression. Nth expressions are allowed only inside some pseudo classes and names of these pseudo classes are not reserved keywords in css.<br />
<br />
Nth expression is an expression of the form <code>an+b</code> where <code>a</code> and <code>b</code> are optional integers. Examples of valid nth expressions: <code>5n+2</code>, <code>-5n-2</code>, <code>-5n</code>, <code>-2</code>, <code>-n</code>, <code>n</code>.<br />
<br />
We wanted our grammar to accept nth expressions, but only as parameters of pseudo classes where it is actually allowed. We wanted it to reject nth expressions as parameters of all remaining pseudo classes.<br />
<br />
All names normally correspond to <code>IDENT</code> tokens. Creating special token corresponding to nth pseudo class name is unpractical, because they are not reserved keywords. For example, they are also perfectly valid class names or element names. Having special token would force us to replace almost all <code>IDENT</code> occurrences by <code>IDENT | NTH</code>.<br />
<br />
Therefore, we are left with general identifier <code>IDENT</code> which can be either normal or nth pseudo class name. Standard syntactical regular expressions are unable to distiguish between them, but semantic predicates can.<br />
<br />
<a href="#SolvingInitialUseCasesKeywordsnth">Link to solution.</a><br />
<br />
<a name="UseCasesSignificantWhitespaces"></a><h5>Significant Whitespaces</h5>Css has semi-important whitespaces. Semi-important means that most of them represent only ends of tokens and that is where their usefulness ends. For example, whitespaces in declaration are irrelevant. Following declarations are equal: <pre class="brush:css">padding : 2;
padding:2;
</pre><br />
Most of CSS grammar behave the above way, so there is strong temptation to throw all whitespaces away. However, if we do that, then the next two selectors end up as the same <code>IDENT COLON IDENT LPAREN COLON IDENT RPAREN COLON IDENT LPAREN COLON IDENT RPAREN LBRACE</code> token stream: <pre class="brush:css">div :not(:enabled) :not(:disabled) {}
div:not(:enabled):not(:disabled) {}
</pre><br />
Whitespaces in selectors are significant. The first selector is equivalent to <code>div *:not(:enabled) *:not(:disabled)</code> while the second is not. <br />
<br />
Note: <a href="http://www.antlr.org/grammar/1240941192304/css21.g">CSS 2.1 grammar</a> available from antlr site ignores this issue. If you want to use it, you have to fix it first.<br />
<br />
One solution would be to stop hiding whitespaces. This would force us to add explicit whitespace handing <code>WS*</code> into all possible places of all parser rules. That would be a lot of work and the resulting grammar would be less readable.<br />
<br />
It is also possible to give up on selectors tree building in antlr parser and write custom hand made tree builder for it. This is how we did it originally and we can safely tell that it works, but requires more time and debugging then the final semantic predicates based solution.<br />
<br />
<a href="#SolvingInitialUseCasesSignificantWhitespaces">Link to solution.</a><br />
<br />
<a name="Basics"></a><h4>Basics</h4>We start with semantic predicates syntax and some needed terminology. This chapter also outlines basics of what happens if predicate fails. We will not go into details, those are described in the next chapter.<br />
<br />
<a name="BasicsSyntax"></a><h5>Syntax</h5>Semantic predicate is always enclosed inside curly braces followed either by question mark or question mark and double arrow:<ul><li><code>{ condition }?</code></li>
<li><code>{ condition }?=></code></li>
</ul><br />
First example uses simple <code>1+2==3</code> and <code>2+3==5</code> conditions. The grammar is stored inside the Basics.g file:<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: LETTER {1+2==3}? LETTER;
NUMERAL: '0'..'9';
number: {2+3==5}?=> NUMERAL NUMERAL;
</pre><br />
<a name="BasicsTerminology"></a><h5>Terminology</h5>Depending on which syntax is used and where it is placed, semantic predicates can be called by one of three different names:<br />
<ul><li>disambiguating semantic predicate,</li>
<li>validating semantic predicate,</li>
<li>gated semantic predicate.</li>
</ul><br />
<a name="BasicsTerminologyDisambiguatingSemanticPredicate"></a><h6>Disambiguating Semantic Predicate</h6>Disambiguating predicates use the shorter <code>{...}?</code> syntax. However, a predicate is called disambiguating only if it is placed in the beginning of a rule or in the beginning of an alternative.<br />
<br />
Disambiguating semantic predicate:<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
// beginning of a rule
rule: {1+2==3}? LETTER LETTER;
// beginning of an alternative
alternatives: LETTER (
{2+3==5}? LETTER*
| {2+3==5}? LETTER+
);
</pre><br />
<a name="BasicsTerminologyValidatingSemanticPredicate"></a><h6>Validating Semantic Predicate</h6>Validating predicates also use the shorter <code>{...}?</code> syntax. The difference against disambiguating predicates is only in placement. Validating predicates are placed in the middle of a rule or in the middle of an alternative:<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: LETTER {1+2==3}? LETTER;
</pre><br />
<a name="BasicsTerminologyGatedSemanticPredicate"></a><h6>Gated Semantic Predicate</h6>Gated semantic predicates use the longer <code>{...}?=></code> syntax. The condition can be placed anywhere.<br />
<br />
Gated semantic predicate:<pre class="brush:java">NUMERAL: '0'..'9';
number: {2+3==5}?=> NUMERAL NUMERAL;
</pre><br />
<a name="BasicsFailedPredicates"></a><h5>Failed Predicates</h5>As we explained in <a href="http://meri-stuff.blogspot.sk/2011/09/antlr-tutorial-expression-language.html#ParserBasics">expression language tutorial</a> post, parser starts knowing which rule should correspond to the input and then tries to match it to the input. Matching always starts from left-most element of the rule and continues to the right. <br />
<br />
If matching encounters semantic predicate, then it tests whether the condition is satisfied. If it is not satisfied, <code>FailedPredicateException</code> exception is thrown.<br />
<br />
Consider Basics.g grammar shown at the beginning of this chapter:<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: LETTER {1+2==3}? LETTER;
NUMERAL: '0'..'9';
number: {2+3==5}?=> NUMERAL NUMERAL;
</pre><br />
If you open generated <code>BasicsParser</code> class, you will find that each rule has corresponding method with the same name. Both of them contains a copy of the predicate and both of them throws an exception if the condition is not satisfied:<pre class="brush:java">// inside the generated word() method
if ( !((1+2==3)) ) {
throw new FailedPredicateException(input, "word", "1+2==3");
}
// inside the generated number() method
if ( !((2+3==5)) ) {
throw new FailedPredicateException(input, "number", "2+3==5");
}
</pre><br />
Prediction, e.g. what happens if the parser encounters rule with both multiple alternatives and predicates, for example <code>start: word | number</code> rule, is described in the next chapter.<br />
<br />
<a name="HoistingandPrediction"></a><h4>Hoisting and Prediction</h4>Depending on where and how you use semantic predicates, the parser may try to avoid failed predicate exceptions. Used strategy is called "hoisting" and it is what makes predicates useful. <br />
<br />
This chapter explains what hoisting is and what consequences it has. Then we will explain when it is used and when it is not used. <br />
<br />
<a name="HoistingandPredictionWhatItIs"></a><h5>What It Is</h5>A parser that encountered rule with multiple alternatives has to decide which of these alternatives should be used. If some of them start with a predicate, parser may use that predicate to help with the decision. <br />
<br />
Consider the grammar stored in DisambiguatingHoistingNeeded.g file:<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: {1+2==3}? LETTER LETTER;
sameWord: {1+2!=3}? LETTER LETTER;
start: word | sameWord;
</pre><br />
Both <code>word()</code> and <code>sameWord()</code> methods of the generated parser contains the usual failed predicate check. <code>DisambiguatingHoistingNeededParser</code> class extract:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">//inside the word() method
if ( !((1+2==3)) ) {
throw new FailedPredicateException(input, "word", "1+2==3");
}
//inside the sameWord() method
if ( !((1+2!=3)) ) {
throw new FailedPredicateException(input, "sameWord", "1+2!=3");
}
</pre><br />
</div></div>In addition, the code corresponding to the <code>start</code> rule contains copies of both <code>word</code> and <code>sameWord</code> semantic predicates. The part that chooses which rule to use next contains following code (comments are mine):<br />
<pre class="brush:java">int LA1_2 = input.LA(3);
//predicate copied from the word rule
if ( ((1+2==3)) ) {
alt1=1;
} //predicate copied from the sameWord rule
else if ( ((1+2!=3)) ) {
alt1=2;
}
else {
NoViableAltException nvae = new NoViableAltException("", 1, 2, input);
throw nvae;
}
</pre><br />
The act of copying the predicate into prediction part of the generated parser is called hoisting. <br />
<br />
<a name="HoistingandPredictionConsequences"></a><h5>Consequences</h5>If there would be no hoisting, predicates would act as assertions only. We could use them to validate some conditions and that would be it. The above grammar would be illegal - it has two syntactically equivalent alternatives if you ignore predicates.<br />
<br />
As hoisting copies predicates all over the grammar, it has also several limiting consequences for them. It is not just something that happens on the background you can safely ignore:<ul><li>each predicate can run several times,</li>
<li>order in which predicates run may be hard to predict,</li>
<li>local variables or parameters may not be available in hoisted copies.</li>
</ul><br />
Conditions must be without side effects, repeatable and their evaluation order should not matter. If they are hoisted into other rules, then they can not reference local variables or parameters. <br />
<br />
Only predicates placed in the beginning of a rule are hoisted into other rules. Hoisting in the case of alternatives is only within the rule. Therefore, you can break the third rule if the predicate is not placed in the beginning of the rule.<br />
<br />
<a name="HoistingandPredictionWhenItIsUsed"></a><h5>When It Is Used</h5>Hoisting is used only when the parser has to decide between multiple rules or alternatives and some of them begin with a predicate. If it is gated predicate e.g., condition inside the <code>{...}?=></code> syntax, then the predicate is hoisted no matter what. <br />
<br />
If it is disambiguating predicate e.g., condition inside the <code>{...}?</code> syntax, then the predicate is hoisted only if it is actually needed. The term "actually needed" means that multiple alternatives could match the same input. Otherwise said, it is used only if multiple alternatives are ambiguous for some input.<br />
<br />
Predicates placed in the middle of rules or in the middle of alternatives are never hoisted.<br />
<br />
<a name="HoistingandPredictionWhenItIsUsedIfNeededHoistingDisambiguatingPredicate"></a><h6>If Needed Hoisting - Disambiguating Predicate</h6>Consider the rule <code>start</code> in the DisambiguatingHoistingNotNeeded.g grammar:<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
NUMBER : '0'..'9';
word: {1+2==3}? LETTER LETTER;
differentWord: {1+2!=3}? LETTER NUMBER;
start: word | differentWord;
</pre><br />
The rule <code>start</code> has to choose between the <code>word</code> and <code>differentWord</code> rules. Both of them start with predicate, but the predicate is not needed in order to differentiate between them. The second token of the <code>word</code> is <code>LETTER</code> while the second token of the <code>differentWord</code> is <code>NUMBER</code>. <br />
<br />
Hoisting will not be used. Instead, the grammar will look into upcoming 2 tokens to distinguish between these rules. To verify, open the <code>start()</code> method of the generated <code>DisambiguatingHoistingNotNeededParser</code> class in our sample project: neither <code>1+2==3</code> nor <code>1+2!=3</code> condition was copied into it. <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">int alt1=2;
switch ( input.LA(1) ) {
case LETTER: {
switch ( input.LA(2) ) {
case LETTER: {
alt1=1;
}
break;
case NUMBER: {
alt1=2;
}
break;
default:
NoViableAltException nvae =
new NoViableAltException("", 1, 1, input);
throw nvae;
}
</pre><br />
</div></div>On the other hand, consider the rule <code>start</code> in the DisambiguatingHoistingNeeded.g grammar:<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: {1+2==3}? LETTER LETTER;
sameWord: {1+2!=3}? LETTER LETTER;
start: word | sameWord;
</pre><br />
The rule <code>start</code> has to choose between the <code>word</code> and <code>sameWord</code> rules. These two rules match exactly the same sequence of tokens and differ only by the predicate. <br />
<br />
Hoisting will be used. To verify, open the <code>start()</code> method of the generated <code>DisambiguatingHoistingNeededParser</code> class in our sample project. It contains copies of both <code>1+2==3</code> and <code>1+2!=3</code> conditions. <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">int alt1=2;
switch ( input.LA(1) ) {
case LETTER: {
switch ( input.LA(2) ) {
case LETTER: {
int LA1_2 = input.LA(3);
if ( ((1+2==3)) ) {
alt1=1;
} else if ( ((1+2!=3)) ) {
alt1=2;
} else { /* ... */ }
}
break;
default: // ...
}
}
break;
default: // ...
}
</pre><br />
</div></div>The exactly same thing is happening with disambiguating predicates in alternatives.<br />
<br />
This will not be hoisted (DisambiguatingHoistingNotNeeded.g grammar):<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
alternatives: LETTER (
{2+3==5}? LETTER
| {2+3==5}? NUMBER
);
</pre><br />
This will be hoisted (DisambiguatingHoistingNeeded.g grammar):<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
alternatives: LETTER (
{2+3==5}? LETTER
| {2+3==5}? LETTER
);
</pre><br />
<a name="HoistingandPredictionWhenItIsUsedAlwaysHoistingGatedRules"></a><h6>Always Hoisting - Gated Rules</h6>Look at the <code>start</code> rule in the GatedHoisting.g grammar:<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
NUMBER: '0'..'9';
word: {1+2==3}?=> LETTER LETTER;
differentWord: {1+2!=3}?=> LETTER NUMBER;
start: word | differentWord;
</pre><br />
The rule <code>start</code> has to choose between the <code>word</code> and <code>differentWord</code> words. Both of them starts with predicate and that predicate is not needed in order to differentiate between them. <br />
<br />
However, hoisting will be used because we used gated semantic predicate. To verify, open the <code>start()</code> method of the generated <code>GatedHoisting</code> class in our sample project. It contains copies of both <code>1+2==3</code> and <code>1+2!=3</code> conditions. <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">int LA1_0 = input.LA(1);
if ( (LA1_0==LETTER) && (((1+2==3)||(1+2!=3)))) {
int LA1_1 = input.LA(2);
if ( (LA1_1==LETTER) && ((1+2==3))) {
alt1=1;
}
else if ( (LA1_1==NUMBER) && ((1+2!=3))) {
alt1=2;
}
else {
NoViableAltException nvae =
new NoViableAltException("", 1, 1, input);
throw nvae;
}
}
else {
NoViableAltException nvae =
new NoViableAltException("", 1, 0, input);
throw nvae;
}
</pre><br />
</div></div>The exactly same thing is happening with gated predicates in alternatives. This will be hoisted (GatedHoisting.g grammar):<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
NUMBER: '0'..'9';
alternatives: LETTER (
{2+3==5}?=> LETTER
| {2+3==5}?=>NUMBER
);</pre><br />
<a name="HoistingandPredictionWhenItIsUsedNeverHoistingMiddleofaRule"></a><h6>Never Hoisting - Middle of a Rule</h6>Hoisting is never used if the predicate is located in the middle of a rule or an alternative. It does not matter which predicate type is used. Therefore, if your rules differ only by the predicate, that predicate must be placed in the beginning of a rule or an alternative.<br />
<br />
Non-hoisted gated predicate (GatedNoHoisting.g):<pre class="brush:java">LETTER: 'a'..'z' | 'A'..'Z';
NUMBER: '0'..'9';
//gated predicate in the middle of a rule
word: LETTER {1+2==3}?=> LETTER;
differentWord: LETTER {1+2!=3}?=> NUMBER;
start: word | differentWord;
</pre><br />
Another non-hoisted gated predicate (GatedNoHoisting.g):<pre class="brush:java">LETTER: 'a'..'z' | 'A'..'Z';
NUMBER: '0'..'9';
//gated predicate in the middle of an alternative
alternatives: LETTER (
LETTER {2+3==5}?=> LETTER
| LETTER {2+3==5}?=> NUMBER
);
</pre><br />
Generated parser is in <code>GatedNoHoistingParser</code> class. <br />
<br />
The most important point is that if your rules differ only by the predicate and that predicate is placed in the middle of a rule, antlr will refuse to generate corresponding parser. Next expandable box contains several examples of syntactically incorrect grammars along with antr errors they cause. <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
Incorrect grammar (SyntacticallyIncorrect.g):<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: LETTER {1+2==3}? LETTER;
sameWord: LETTER {1+2!=3}? LETTER;
start: word | sameWord;
</pre><br />
Error in console: <pre class="brush:java" style="text-align: left;">warning(200): org\meri\antlr\predicates\SyntacticallyIncorrect.g:28:6: Decision can match input such as "LETTER LETTER" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
error(201): org\meri\antlr\predicates\SyntacticallyIncorrect.g:28:6: The following alternatives can never be matched: 2
</pre><br />
Another incorrect grammar (SyntacticallyIncorrect.g):<pre class="brush:java">alternativesStart: LETTER (
LETTER {1+2==3}?
| LETTER {1+2!=3}?
);
</pre><br />
Error in console: <pre class="brush:java" style="text-align: left;">warning(200): org\meri\antlr\predicates\SyntacticallyIncorrect.g:31:27: Decision can match input such as "LETTER" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
error(201): org\meri\antlr\predicates\SyntacticallyIncorrect.g:31:27: The following alternatives can never be matched: 2
</pre><br />
Yet another incorrect grammar (SyntacticallyIncorrect.g):<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
gatedWord: LETTER {1+2==3}?=> LETTER;
gatedSameWord: LETTER {1+2!=3}?=> LETTER;
gatedStart: gatedWord | gatedSameWord;
</pre><br />
Error in console: <pre class="brush:java" style="text-align: left;">warning(200): org\meri\antlr\predicates\SyntacticallyIncorrect.g:40:11: Decision can match input such as "LETTER LETTER" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
error(201): org\meri\antlr\predicates\SyntacticallyIncorrect.g:40:11: The following alternatives can never be matched: 2
</pre><br />
Last incorrect grammar (SyntacticallyIncorrect.g):<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
gatedAlternativesStart: LETTER (
LETTER {1+2==3}?=> LETTER
| LETTER {1+2!=3}?=> LETTER
);
</pre><br />
Error in console: <pre class="brush:java" style="text-align: left;">warning(200): org\meri\antlr\predicates\SyntacticallyIncorrect.g:43:32: Decision can match input such as "LETTER LETTER" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
error(201): org\meri\antlr\predicates\SyntacticallyIncorrect.g:43:32: The following alternatives can never be matched: 2
</pre><br />
</div></div><a name="HoistingandPredictionWhenItIsUsedNuances"></a><h5>Nuances</h5>Previous "When It Is Used" sub-chapter shown how predicates behave in clearly hoisted and clearly non-hoisted situations. We selected examples to show as clear and simple situations as possible. <br />
<br />
This sub-chapter contains different set of examples. We picked most potentially confusing we have been aware of. All used examples are located in Nuances.g file.<br />
<br />
<a name="HoistingandPredictionWhenItIsUsedNuancesDisambiguatingPredicatesAdvanced"></a><h6>Disambiguating Predicates - Advanced</h6>Hoisted disambiguating predicates are used only if multiple alternatives are ambiguous for current input. Otherwise said, hoisted copy of the predicate is run only if the actual input could be parsed by multiple alternatives.<br />
<br />
Example: alternatives in the following rule are not syntactically equivalent, because they do not match the same set of inputs. First alternative matches exactly two letters and second alternative matches any number of letters:<br />
<pre class="brush:java">advancedDisambiguating: LETTER (
{1+2==3}? LETTER LETTER
| {1+2!=3}? LETTER*
);
</pre><br />
If the input starts with exactly one <code>LETTER</code>, then it can not be parsed by the first alternative. As only second alternative matches it, predicate will not be used. Parser will use second alternative and if <code>1+2!=3</code> condition happen to be <code>false</code>, parser will throw failed predicate exception.<br />
<br />
However, if the input starts with two letters, then it could be matched by both alternatives and predicate will be used. This is how generated code looks:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: left;"><br />
<pre class="brush:java">int alt4=2;
switch ( input.LA(1) ) {
case LETTER: {
switch ( input.LA(2) ) {
case LETTER: {
int LA4_3 = input.LA(3);
//predicate is used only if first two tokens are LETTER
if ( ((1+2==3)) ) {
alt4=1;
}
else if ( ((1+2!=3)) ) {
alt4=2;
}
else {
// ... irrelevant code ...
}
}
break;
//if the second token is not LETTER, predicate is not used
case EOF: { alt4=2; } break;
default: // ...
}
}
break;
//if the first token is not LETTER, predicate is not used
case EOF: // ...
default: // ...
}
</pre><br />
</div></div>Compare it to very similar gated rule:<br />
<pre class="brush:java">compareGated: LETTER (
{1+2==3}?=> LETTER LETTER
| {1+2!=3}?=> LETTER*
);
</pre><br />
Parser will use the predicate no mater what. The second alternative will never be entered, because the predicate <code>1+2!=3</code> is never satisfied: <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: left;"><br />
<pre class="brush:java">int alt6=2;
int LA6_0 = input.LA(1);
if ( (LA6_0==LETTER) && (((1+2==3)||(1+2!=3)))) {
int LA6_1 = input.LA(2);
if ( (LA6_1==LETTER) && (((1+2==3)||(1+2!=3)))) {
int LA6_3 = input.LA(3);
</pre><br />
</div></div>Gated predicate causes antlr to throw different kind of exception in this case. As we will show <a href="#WritingConditionsLabelsandListsLabelListExample">later in this post</a>, different hoisting of gated and disambiguating predicates can make much bigger difference with more complicated predicates. Namely, it can make difference between accepting and rejecting the input.<br />
<br />
<a name="HoistingandPredictionWhenItIsUsedNuancesLoops"></a><h6>Loops</h6>Although it does not look like that at the first sight, loops are alternatives too. They use prediction to guess whether they should do one more round or whether they should end.<br />
<br />
Use predicate to stay in the loop only while it returns true:<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
loop: ( {somePredicate()}?=> LETTER )*;
</pre><br />
The <code>loop</code> rule will match letters until the function <code>somePredicate()</code> returns <code>false</code> or until the rule runs out of <code>LETTER</code> tokens.<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java" style="text-align: left;">loop1:
do {
int alt1=2;
int LA1_0 = input.LA(1);
// predicate is used during the prediction
if ( (LA1_0==LETTER) && ((somePredicate()))) {
alt1=1;
}
//matching: either jump out or match another LETTER
switch (alt1) {
case 1: {
if ( !((somePredicate())) ) {
throw new FailedPredicateException(...);
}
// ... match LETTER ...
}
break;
default:
break loop1;
}
} while (true);
</pre><br />
</div></div>Disambiguating predicate can not be used for this purpose. Next predicate will not be used to decide whether parser should stay in the loop or not:<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
loopDisambiguating: ( {somePredicate()}? LETTER )*;
</pre><br />
Technically, the loop is deciding between <code>LETTER</code> and <code><nothing></code> alternatives. Those are syntactically different and prediction uses disambiguating predicates only if it has to decide between syntactically ambiguous alternatives.<br />
<br />
The <code>loopDisambiguating</code> rule will match letters until it runs out of <code>LETTER</code> tokens. If the function <code>somePredicate()</code> returns <code>false</code> during that time, the rule will throw <code>FailedPredicateException</code> exception.<br />
<br />
Generated code is very similar to the previous one, only the prediction part changes. Predicate is not used:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java" style="text-align: left;">loop2:
do {
int alt2=2;
//prediction ignores the predicate
switch ( input.LA(1) ) {
case LETTER: {
alt2=1;
}
break;
}
//matching: either jump out or match another LETTER
switch (alt2) {
case 1: {
if ( !((somePredicate())) ) {
throw new FailedPredicateException(...);
}
// ... match LETTER ...
}
break;
default:
break loop2;
}
} while (true);
</pre><br />
</div></div><a name="HoistingandPredictionWhenItIsUsedNuancesUncoveredAlternatives"></a><h6>Uncovered Alternatives</h6>It is perfectly ok to leave some alternatives uncovered. Alternatives with predicates will work as expected. Gated predicate is always hoisted and disambiguated predicate is hoisted only if there are multiple ambiguous alternatives.<br />
<br />
Gated predicate is always hoisted:<br />
<pre class="brush:java">uncoveredGated: LETTER (
{3+4==7}?=> LETTER
| NUMBER
);
</pre><br />
Hoisted disambiguated predicate:<br />
<pre class="brush:java">uncoveredDisambiguatingHoisted: LETTER (
{2+5==7}? LETTER*
| LETTER+
);
</pre><br />
Non hoisted disambiguated predicate:<br />
<pre class="brush:java">uncoveredDisambiguatingNotHoisted: LETTER (
{2+4==6}? LETTER
| NUMBER
);
</pre><br />
<a name="HoistingandPredictionWhenItIsUsedNuancesCombinationofGatedandDisambiguatedPredicates"></a><h6>Combination of Gated and Disambiguated Predicates</h6>If one alternative is gated and the other is disambiguated, then gated predicate is always hoisted and disambiguated predicate is hoisted only if it is actually needed.<br />
<br />
Gated predicate is hoisted while disambiguated predicate is not hoisted:<br />
<pre class="brush:java">combinationDisambiguatingNotHoisted: LETTER (
{1+4==5}?=> LETTER
| {1+4!=5}? NUMBER
);
</pre><br />
Both predicates are hoisted:<br />
<pre class="brush:java">combinationDisambiguatingHoisted: LETTER (
{1+5==6}?=> LETTER*
| {1+5!=6}? LETTER+
);
</pre><br />
<a name="HoistingandPredictionWhenItIsUsedNuancesAdditionalParenthesis"></a><h6>Additional Parenthesis</h6>If you close disambiguating predicate in parenthesis, the predicate is still be treated like disambiguating predicate.<br />
<br />
Another way to write disambiguating predicate:<br />
<pre class="brush:java">stillDisambiguating: ({2+2==4}?) LETTER;
testStillDisambiguating: stillDisambiguating | LETTER;
</pre><br />
If you put additional parenthesis around gated predicate, the predicate will be ignored:<br />
<pre class="brush:java">ignored: ({3+3==6}?)=>LETTER;
</pre><br />
<a name="HoistingandPredictionWhenItIsUsedNuancesBacktracking"></a><h4>Backtracking</h4>Predicates run even if the parser is backtracking e.g, if it is inside syntactical predicate. If the parser is backtracking and the predicate fails, backtracking fails too. Failed predicate exception is thrown only if the parser is not backtracking.<br />
<br />
The <code>backtrack</code> rule initiates backtracking (Backtracking.g):<br />
<pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: LETTER {1+2==3}? LETTER;
number: LETTER {2+3!=5}? LETTER;
backtrack: (number)=> number | word;
</pre><br />
Since backtracking is possible, generated predicate check is different:<br />
<pre class="brush:java">if ( !((2+3!=5)) ) {
if (state.backtracking>0) {state.failed=true; return retval;}
throw new FailedPredicateException(input, "number", "2+3!=5");
}
</pre><br />
Backtracking is yet another reason why conditions must not have side effects, must be repeatable and their evaluation order must not matter. <br />
<br />
<a name="WritingConditions"></a><h4>Writing Conditions</h4>This chapter shows how to write advanced conditions for semantic predicates. First, we will show how to access and use input token stream. We will also explain how to reference labeled tokens. Last part is about local variables in non-hoisted conditions. <br />
<br />
Unless specified otherwise, all used examples are located in Environnement.g file.<br />
<br />
<a name="WritingConditionsInputTokenStream"></a><h5>Input Token Stream</h5>Each generated parser has public <code>TokenStream input</code> field. This field provides access to the whole input token stream and also to current position in that token stream. Its most important method is the <code>Token LT(int k)</code> method. The parameter <code>k</code> contains relative position of the token you are interested in. <br />
<br />
The number 1 means "look ahead one token", 2 means "second token ahead" and so on. Negative numbers reference passed tokens: -1 will return previous token, -2 returns the one before it and so on. Do not use 0. Its meaning is undefined and the default parser returns <code>null</code>. <br />
<br />
Note: relative referencing works correctly even when the grammar is in backtracking state. -1 is always previous token and 1 is always the next token.<br />
<br />
<a name="WritingConditionsInputTokenStreamExamples"></a><h6>Examples</h6>Disambiguating: if the <code>word</code> starts with letter <code>a</code>, then it must have at least two letters: <pre class="brush:java">word: LETTER (
{ input.LT(-1).getText().equals("a")}? LETTER+
| { !input.LT(-1).getText().equals("a")}? LETTER*
);
</pre><br />
Gated: if the second numeral of the <code>number</code> is <code>9</code>, then it must have exactly 3 numerals: <pre class="brush:java">number: NUMERAL (
{input.LT(1).getText().equals("9")}?=> NUMERAL NUMERAL
| {!input.LT(1).getText().equals("9")}?=> NUMERAL*
);
</pre><br />
Note: the choice of predicate <a href="#HoistingandPredictionWhenItIsUsedNuancesDisambiguatingPredicatesAdvanced">slightly matter</a> in this case. It influences what kind of error will be thrown if the input does not match the rule. <br />
<br />
<a name="WritingConditionsLabelsandLists"></a><h5>Labels and Lists</h5>Predicates can reference and use any previously defined label or label list the same way as actions can. <br />
<br />
<a name="WritingConditionsLabelsandListsLabelExample"></a><h6>Label Example</h6>If the first letter of the word is <code>a</code>, then the word must have at least two letters: <pre class="brush:java">labeledWord: a=LETTER (
{ $a.getText().equals("a")}? LETTER+
| { !$a.getText().equals("a")}? LETTER*
);
</pre><br />
<a name="WritingConditionsLabelsandListsLabelListExample"></a><h6>Label List Example</h6>If the word starts with less then 3 letters, then it must end with a number: <pre class="brush:java">labeledListWord: a+=LETTER+ (
{ $a.size() < 3 }?=> NUMERAL
| { $a.size() >= 3}?=>
);
</pre><br />
Note: the choice of predicate does matter in this case. The above example works correctly only if it uses gated <code>{...}?=></code> predicate instead of disambiguating <code>{...}?</code> one. The <code>NUMERAL</code> and <code><nothing></code> are syntactically different. Disambiguating predicate would not be used for prediction e.g., it would not be hoisted. The parser would base its decision solely on the next token (is it <code>NUMERAL</code>?). <br />
<br />
The condition would be used as an afterwards assertion to check whether the number of letters was right. Such grammar would throw an exception on the <code>abcd9</code> input, while ours would accept it. <br />
<br />
<a name="WritingConditionsLabelsandListsUndefinedLabels"></a><h6>Undefined Labels</h6>Predicate can NOT reference not-yet-defined labels. The parser is generated, but the first attempt to use the rule throws null pointer exception in runtime:<pre class="brush:java">//this would cause null pointer exception
nullPointerAtPredicate: LETTER { $a.getText().equals("a") }? a=LETTER;
</pre><br />
<a name="WritingConditionsLabelsandListsLabelsandHoisting"></a><h6>Labels and Hoisting Into Other Rules</h6>As the label has to be defined before being used in the predicate and predicates are copied into other rules only if they are located in the very beginning of a rule, you do not have to worry about hoisting into other rules.<br />
<br />
<a name="WritingConditionsAccesstoLocalVariables"></a><h5>Access to Local Variables</h5>Antlr allows you to define custom local variables and use them withing one rule. If you are sure that predicate will not be copied into other rules, it can use them. Of course, using local variables if the predicate can be copied into other rules will result in faulty parser.<br />
<br />
Create local variables and use them in the predicate. If the word starts with less then 10 letters, then it must end with a number <pre class="brush:java">localVariables
@init {int num=0;} //define local variable num
: (LETTER { num++; })* //raise the num variable by 1 for each letter
( // what should follow depends on the variable value
{ num < 10 }?=> NUMERAL
| { num >= 10}?=>
);
</pre><br />
Note: The same warning as before applies. We must use gated predicates.<br />
<br />
You must be especially careful not to use local variables in potentialy hoisted predicates. For example, Antlr Reference book recommends following rule to match only numbers composed of less then four numerals (ANTLRReference3Error.g): <pre class="brush:java">localVariablesWarning
@init {int n=1;} // n becomes a local variable
: ( {n<=4}?=> NUMERAL {n++;} )+ // enter loop only if n<=4
;
</pre><br />
The above rule works well in isolation, that is when it is not used in other rules. Unfortunately, if you include it into other rules, the predicate may be hoisted into that other rule (ANTLRReference3Error.g):<br />
<pre class="brush:java">// syntax error in generated parser
syntaxError: localVariablesWarning | LETTER;</pre><br />
The <code>n<=4</code> predicate will be copied into the <code>syntaxError</code> rule. The variable <code>n</code> is not accessible inside that rule and generated parser will be syntactically incorrect.<br />
<br />
<a name="SolvingInitialUseCases"></a><h4>Solving Initial Use Cases</h4>Finally, we are going to solve both use cases described in the motivational chapter. <br />
<br />
<a name="SolvingInitialUseCasesKeywordsnth"></a><h5>Keywords - nth</h5><a href="#UseCasesKeywordsnth">Link to original use case.</a><br />
<br />
We created function <code>isInsideNth</code> that returns <code>true</code> only if previous token matched name of some nth pseudo class. The function is used as condition inside gated predicate. Generated parser will assume that input contains nth expression if and only if it is inside nth pseudo class.<br />
<br />
UseCasesNth.g file:<br />
<pre class="brush:java">@parser::members {
private static Set<String> NTH_PSEUDOCLASSES = new HashSet<String>();
static {
NTH_PSEUDOCLASSES.add("nth-child");
NTH_PSEUDOCLASSES.add("nth-last-child");
NTH_PSEUDOCLASSES.add("nth-of-type");
NTH_PSEUDOCLASSES.add("nth-last-of-type");
}
public boolean isInsideNth() {
return isNthPseudoClass(input.LT(-1));
}
private boolean isNthPseudoClass(Token a) {
if (a == null)
return false;
String text = a.getText();
if (text == null)
return false;
return NTH_PSEUDOCLASSES.contains(text.toLowerCase());
}
}
LPAREN: '(';
RPAREN: ')';
COLON: ':';
COMMA: ',';
IDENT : ('a'..'z' | 'A'..'Z')+;
//pseudoparameters and nth with dummy syntax
pseudoparameters: IDENT (COMMA IDENT)*;
nth: IDENT; //real nth syntax ommited for simplicity sake
// pseudoclass
pseudo
: COLON COLON? IDENT ((
{ isInsideNth()}?=> LPAREN nth RPAREN
| LPAREN pseudoparameters RPAREN
)?)
;
</pre><br />
An alternative solution with labels and rewrite rule: <pre class="brush:java">//different solution - note that we need to use rewrite rules in this case
pseudoDifferentSolution
: COLON COLON? name=IDENT ((
{ isNthPseudoClass($name)}?=> LPAREN nthParameters=nth RPAREN
| LPAREN parameters=pseudoparameters RPAREN
)?)
-> $name $nthParameters? $parameters?
;
</pre><br />
<a name="SolvingInitialUseCasesSignificantWhitespaces"></a><h5>Significant Whitespaces</h5><a href="#UseCasesSignificantWhitespaces">Link to original use case.</a><br />
<br />
Css selectors can be composed of multiple parts separated by combinators <code>></code>, <code>+</code>, <code>~</code> and <code><space></code>. Each part called simple selector starts with an optional element name and may be followed by multiple pseudo classes, attributes and similar structures.<br />
<br />
Ignoring the space as combinator problem, simplified simple selector grammar can look like this:<br />
<pre class="brush:java">COLON: ':';
STAR: '*';
NUMBER: ('0'..'9')+;
IDENT : ('a'..'z' | 'A'..'Z')+;
//some options have been removed from following rules for simplicity sake
elementName: IDENT | STAR | NUMBER;
pseudoClass: COLON COLON? IDENT;
elementSubsequent: pseudoClass;
simpleSelectorWrong:
(elementName elementSubsequent*)
| elementSubsequent+
;
</pre><br />
The above <code>simpleSelectorWrong</code> rule matches valid simple selectors: <code>h1</code>, <code>h1:first-child:hover</code>, <code>:first-child:hover</code> and <code>:hover</code>. <br />
<br />
Unfortunately, as whitespaces are thrown away, the above rule matches more then that. For example, it would match also <code>h1:first-child :hover</code> which should be interpreted exactly the same way as <code>h1:first-child *:hover</code> selector e.g., as two simple selectors joined by <code><space></code>.<br />
<br />
We created method that returns true only if there is no hidden token between previous and next tokens. Unless configured otherwise, all tokens are instance of <code>CommonToken</code> class. Since common token knows its start and stop index, we can cast and compare them to see whether there was something between them. <br />
<br />
New parser methods (UseCasesSelectors.g):<br />
</div><pre class="brush:java">@parser::members {
public boolean onEmptyCombinator(TokenStream input) {
return !directlyFollows(input.LT(-1), input.LT(1));
}
private boolean directlyFollows(Token first, Token second) {
CommonToken firstT = (CommonToken) first;
CommonToken secondT = (CommonToken) second;
if (firstT.getStopIndex() + 1 != secondT.getStartIndex())
return false;
return true;
}
}
</pre><div style="text-align: justify;"><br />
Fixed simple selector uses gated predicate to check whether it should or should not continue adding subsequent elements (UseCasesSelectors.g):<br />
</div><pre class="brush:java">simpleSelector: (
elementName ({!onEmptyCombinator(input)}?=>elementSubsequent)*
) | (
elementSubsequent ({!onEmptyCombinator(input)}?=>elementSubsequent)*
);
</pre><div style="text-align: justify;"><br />
We <a href="#HoistingandPredictionWhenItIsUsedNuancesLoops">have to use</a> gated predicates in this case. If we would use disambiguating predicate, generated parser would not use our predicate to decide whether to stay inside the loop or not. It is because the loop is technically deciding between <code>elementSubsequent</code> and <code><nothing></code> alternatives and those are syntactically different. The <code>{...}?</code> predicate would not be used during the prediction, it would just occasionally throw exceptions.<br />
<br />
<a name="WrappingItUp"></a><h4>Wrapping It Up</h4>Semantic predicates are java conditions written inside the grammar. They are copied into generated parser as they are, without any changes. If token matching algorithm reaches semantic predicate and that predicate fails, <code>FailedPredicateException</code> exception is thrown.<br />
<br />
If a rule or an alternative starts with semantic predicate, that semantic predicate can be used during prediction phase. Failed predicates during the prediction phase never throw exceptions, but they may disable some alternatives. This is called hoisting. <br />
<br />
Conditions must be without side effects, repeatable and their evaluation order should not matter. If they are hoisted into other rules, then they can not reference local variables or parameters.<br />
<br />
Semantic predicates are used in three different ways: as validating semantic predicates, as disambiguating semantic predicates and as gated semantic predicates.<br />
<br />
<a name="WrappingItUpValidatingSemanticPredicates"></a><h5>Validating Semantic Predicates</h5>Validating semantic predicates act as assertions only. As a result, validating semantic predicates are never hoisted.<br />
<br />
Condition is closed inside curly braces followed by question mark <code>{ condition }?</code>. It must be placed either in the middle of a rule or in the middle of an alternative: <pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
word: LETTER {1+2==3}? LETTER;
</pre><br />
<a name="WrappingItUpDisambiguatingSemanticPredicates"></a><h5>Disambiguating Semantic Predicates</h5>Disambiguating semantic predicates help to choose between syntactically equivalent alternatives. As a result, disambiguating semantic predicates are hoisted only if the parser has to choose between multiple ambiguous alternatives.<br />
<br />
Disambiguating semantic predicates use exactly the same syntax as validating predicates. Condition is closed inside curly braces followed by question mark <code>{ condition }?</code>. However, they must be placed either in the beginning of a rule or in the beginning of an alternative: <pre class="brush:java">LETTER : 'a'..'z' | 'A'..'Z';
// beginning of an alternative
alternatives: LETTER (
{2+3==5}? LETTER*
| {2+3==5}? LETTER+
);
</pre><br />
<a name="WrappingItUpGatedSemanticPredicates"></a><h5>Gated Semantic Predicates</h5>Gated semantic predicates are used to dynamically turn on and off portions of grammar. As a result, all gated predicates placed in the beginning of a rule or an alternative are hoisted. Gated predicates placed in the middle of a rule or an alternative are never hoisted.<br />
<br />
Condition is closed inside curly braces followed by question mark and double arrow <code>{ condition }?=></code>: <pre class="brush:java">NUMERAL: '0'..'9';
number: {2+3==5}?=> NUMERAL NUMERAL;
</pre><br />
<a name="Resources"></a><h4>Resources</h4><ul><li><a href="https://wincent.com/wiki/ANTLR_predicates">Wincent Wiki</a></li>
<li><a href="http://stackoverflow.com/questions/3056441/what-is-a-semantic-predicate-in-antlr3">Stack Overflow Question</a></li>
<li><a href="http://pragprog.com/book/tpantlr/the-definitive-antlr-reference">The Definitive ANTLR Reference</a></li>
</ul></div><br />
<script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>
Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com2tag:blogger.com,1999:blog-3355333693246614846.post-44128858877987581092012-11-15T00:02:00.001-08:002013-03-31T01:25:36.938-07:00Project Report - Less4j<style>div.blogger-clickTrap {display: none;}</style><br />
<div style="text-align: justify;">Few months ago I decided to create java port of <a href="http://lesscss.org/">less.js</a>. Less is an extension of CSS - it adds constants, mixins, ruleset nesting, namespaces and so on. Both reasons why I picked up less and short less introduction are written in <a href="http://meri-stuff.blogspot.sk/2012/06/inheritance-and-variables-in-css-with.html">introduction into less</a>.<br />
<br />
This report starts with current project status and its possible future directions. Second part recaps development and my experiences with used tools. Finally, the last part sums up why did I started with the project in the first place and whether it was worth it. <a name='more'></a><br />
<br />
<h4>Current Status</h4>Less4j can be used as both library and command line compiler. The library version is available in <a href="http://search.maven.org/#browse%7C1893223923">Maven central</a> and command line compiler can be downloaded from Github <a href="https://github.com/SomMeri/less4j/downloads">downloads</a> page.<br />
<br />
The project is <a href="https://github.com/SomMeri/less4j#readme">hosted</a> on Github. Github provides also a simple issue <a href="https://github.com/SomMeri/less4j/issues?state=open">tracking</a> system and extremely simple <a href="https://github.com/SomMeri/less4j/wiki">wiki</a> for documentation. Less4j uses also hosted <a href="http://meri-stuff.blogspot.sk/2012/10/travis-ci-continuous-integration-server.html">continuous integration</a> named Travis.<br />
<br />
The biggest language features are already there, at least in some simple version:<br />
<ul><li>variables,</li>
<li>operations,</li>
<li>nesting,</li>
<li>mixins,</li>
<li>namespaces.</li>
</ul><br />
Just as less.js, less4j keeps comments on place. Translation will not remove them. <br />
<br />
Remaining features should be doable without major rewrite. There are only two exceptions: <br />
<ul><li><a href="https://github.com/SomMeri/less4j/issues/21">Escaping</a> which allows to insert any non-parseable input into less. </li>
<li><a href="https://github.com/SomMeri/less4j/issues/46">Browser hacks</a> which allows to write browser specific css. They take advantage of various browsers bugs and theoretically should not be used. Practically, they are often useful and sometimes unavoidable. </li>
</ul>Both will shake things a bit for sure, but since I have not thought about them too much yet, it is hard to make any guesses on how much.<br />
<br />
I try to have the <a href="https://github.com/SomMeri/less4j/wiki">documentation</a> for all features completed at the release time. It captures what I intended to implement and how less language works. The idea is to have all potentially confusing points written out and to give warning on things that may change in the future.<br />
<br />
<h4>Future</h4>Less.js have big plans for future. First, they plan to introduce backward incompatible changes in the 1.4.0 version. They also plan to introduce language configuration options: you will be able to turn certain syntaxes on and off.<br />
<br />
<h5>Backward Incompatible Changes</h5>They have very solid reasons for all planned backward incompatible changes. New syntax will be a bit more strict because current looser rules occasionally cause problems. Changes will remove some "tricky and subtle" situations from the language.<br />
<br />
I was tempted to chase directly the future 1.4.0 version. In addition to previously stated arguments, it would be better if new language users would use new syntax and it is hard to guess how long will less.js support the old style.<br />
<br />
On the other hand, it is not entirely sure how it will look when it will be done. The discussions are still going on:<br />
<ul><li><a href="https://github.com/cloudhead/less.js/issues/146">issue 146,</a></li>
<li><a href="https://github.com/cloudhead/less.js/issues/701">issue 701,</a></li>
<li><a href="https://github.com/cloudhead/less.js/issues/915">issue 915,</a></li>
<li><a href="https://github.com/cloudhead/less.js/issues/1007">issue 1007.</a></li>
</ul><br />
<h5>Optional Syntaxes</h5>Optional syntaxes are quite interesting challenge, but I still have to see whether they make sense for the compiler port. They are not priority for now. It is more important to get at least one version of the language done.<br />
<br />
<h4>Experiences</h4>This chapter contains my experiences and opinions on tools used in the project. <br />
<br />
<h5>Porting Language</h5>Writing core was much faster and easier then I expected. Two things greatly helped to start the development:<br />
<ul><li>Less.js has unit tests I could take and use.</li>
<li>A <a href="http://www.antlr.org/grammar/1240941192304/css21.g">basic CSS 2.1</a> grammar is available on antlr site.</li>
</ul><br />
Granted, the grammar taken from antlr site has some bugs, so reserve few days for testing and debugging if you plan to use it. It also contains only CSS 2.1, version 3 features have to added. However, those are only minor problems. Especially considering that w3c <a href="https://github.com/SomMeri/less4j/wiki/W3C-CSS3-Specification-Compatibility">published also tests</a> for major parts of css specification.<br />
<br />
Less language turned out to be much more elusive then I expected. Every major language feature has a lot of undocumented sub-features. Every time I think that the feature is finished, I end up finding that less.js can actually do much more. Although it feels like all majors features are already there, I'm sure that it is just because I do not know them all. <br />
<br />
Less documentation is sparse and the only way to learn about features is to read discussions in their issue database. Of course that has some benefits too: their issue database is a great resource of potential tests and edge use cases. Plus, the ability to read past discussions on features helps me understand and appreciate less.js design decisions.<br />
<br />
<h5>Learning Compilers</h5>I tried to watch coursera compilers lectures in the beginning, but they did not seemed to help me. <br />
<br />
"<a href="http://pragprog.com/book/tpdsl/language-implementation-patterns">Pragmatic language implementation patterns</a>" book turned out to be more interesting and useful. It is well written and has a short chapter on language to language translation. I guess that "<a href="http://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_and_Tools">Compilers - Principles, Techniques, and Tools</a>" could also contain something useful, but I did not read it.<br />
<br />
I happen to use design not recommend by the pragmatic book and I do understand why they do not recommend it by now. Less4j creates abstract syntax tree for less and then transforms it into abstract syntax tree for css. The book does not explain why it is not ideal, so here it goes: it is more error prone then building ad hoc structures they recommend and it occasionally requires more work. However, I still think that less4j is small enough and less is close enough to css to make these disadvantages small.<br />
<br />
<h5>Antlr</h5>I puzzled over antlr few times, but its help was big anyway. I guess I would still be in the css parsing phase without it.<br />
<br />
Most troubles have been caused by the fact that css is very loose language and by my use of semantic predicates (java conditions embedded into grammar). First, specifying grammar for strict language would be easier almost by definition. Second, semantic predicates tend to confuse antlr generation phase. Antlr then sees multiple ways to parse the same input and tend to suppress wrong alternatives on such inputs. <br />
<br />
Of course, I could imagine also better antlr-maven-eclipse integration, the one that would not occasionally hide errors and warnings from me. Of course, I also could fix that one by myself if it would really bothered me.<br />
<br />
I would also welcome callbacks on rules entering and leaving. It would be great if I could have a method to be called each time the parser leaves some rule. I do know about the <code>after</code> keyword, but that one has to be copied after each rule separately and is ignored in case of parse error.<br />
<br />
<h5>Maven Release</h5>Releasing the library into Maven central for the first time takes some time, but any subsequent release is a question of two commands. The release plugin creates a tag, increases version in pom.xml, commits changes into repository, builds the library, gpg signs it and uploads it into maven central. All that remains is to sign into Sonatype nexus and click two buttons to actually release it. Simple and fast.<br />
<br />
First release is a bit longer. First, you have to configure pom.xml for <a href="https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide">release</a> and make sure that you project follows <a href="https://docs.sonatype.org/display/Repository/Central+Sync+Requirements">Sonatype guidelines</a>. Then there is some bureaucracy: create and publish gpg key, create an account and open issue in Sonatype tracker. All new projects are checked by a human and if your project does not follow guidelines, you will have to change it.<br />
<br />
All along with examples is described in <a href="http://datumedge.blogspot.sk/2012/05/publishing-from-github-to-maven-central.html">this blog post</a>. I would just add that gpg signing plugin has a bug and is unable to ask for gpg passphrase. You have to add the switch <code>-Dgpg.passphrase=<passphrase></code> to the <code>mvn release:perform</code> command to make it run.<br />
<br />
<h5>Github</h5>Github was made for coding and it shows. It has great code collaboration tools. It can show colored diff for any commit and anyone can comment on any code line. It provides all kind of stats about commit frequencies and sizes. Do not forget the integrated hosted continuous integration (provided by different company), it is also very useful. And so on, the more people collaborate on commits, the better Github is.<br />
<br />
I would welcome also a bit more elaborate issue tracking system, more powerful wiki and more customizable notification system. Issue can be opened and closed, can have labels and that is it. It would be useful to have more bug statuses, special field for priority/severity, division on bugs and features and so on. I would use them if I would have the choice. While it is possible to use labels for that purpose, I would prefer not having to do it.<br />
<br />
Wiki is clearly not meant for big documentation. Syntax wise, it supports 10 different syntaxes, the default one being markdown. On the other hand, it is impossible to generate table of contents. It does not support refactorable page linking. Ok, I'm not sure whether there is a system with such support. But, those other systems can automatically look for all broken cross links. While it is understandable that Github does not allow javascript, I still miss it.<br />
<br />
Projects search on Github is also very simple, but I do not care about that when writing a project.<br />
<br />
<h4>Conclusion</h4>This chapter contains overall accounting: was it worth the time and effort? The first part explains why I started less4j in the first place and what I was expecting out of that. The rest sums up benefits and cons of the project.<br />
<br />
<h5>Motivation</h5>Bringing less.js into java is easy even without this port. There are already <a href="http://meri-stuff.blogspot.sk/2012/06/inheritance-and-variables-in-css-with.html#ServerSideIntegration">tools</a> that use Rhino to <a href="http://meri-stuff.blogspot.sk/2012/08/wro4j-page-load-optimization-and-lessjs.html">comfortably integrate</a> it into Java projects. Strictly speaking, porting the thing was not necessary. It is usable as it is. <br />
<br />
On the other hand, writing the compiler would allow me to learn more about the language I was starting to like. It would also allow me to learn more about ANTLR and to use in a real world non-trivial project. I also liked the idea of coding a project potentially useful to other people too. <br />
<br />
On the less egoistic side, I also through that using native java implementation could have some advantages - some people prefer to use "native" tools. It could have more comfortable API, error reporting and the parser could be used for other purposes later on. For example, I could use it to write a Less Eclipse plugin.<br />
<br />
Lastly, I would be in a great position to document tricky aspects of the language. After all, I would encounter most of them during the development.<br />
<br />
<h5>Personal Pros/Cons</h5>Most important project disadvantages:<br />
<ul><li>It took my attention away from this blog, especially:<ul><li>I did not wrote Eclipse plugins part 2 yet.</li>
</ul></li>
<li>The project is highly addictive.</li>
</ul><br />
Most important project benefits:<br />
<ul><li>The project is highly addictive.</li>
<li>Everything else, especially: <ul><li>It is highly enjoyable coding.</li>
<li>Tried ANTLR on something real.</li>
<li>Learned a lot about less.</li>
</ul></li>
</ul><br />
<h5>Final Conclusion</h5>Definitely worth it.</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com6tag:blogger.com,1999:blog-3355333693246614846.post-87196871199429144432012-10-01T01:00:00.000-07:002012-10-01T01:00:14.749-07:00Travis-CI - Continuous Integration Server<style type="text/css">
div.blogger-clickTrap {display: none;}
</style><div style="text-align: justify;"><a href="http://travis-ci.org/">Travis-CI</a> is hosted continuous integration server for Github projects. The continuous integration part is fairly standard: it runs unit tests and report results after each project change. <br />
<br />
The hosting part is unusual: the service runs tests on their own infrastructure and provides access to test results. The user does not need his own hardware nor to install and run the server. All that is part of the service. <br />
<br />
Any public repository hosted on Github can use Travis-CI for free. Private repositories have to <a href="http://beta.travis-ci.com/">pay</a> and the <a href="http://about.travis-ci.org/docs/user/travis-pro/">Travis Pro</a> version they get is a bit different from the regular one. <br />
<br />
This post is mainly about the free version. First chapter describes what Travis-CI does. Second part contains few things about Travis-CI server, infrastructure and organization behind it. Mini review with our experiences is located in the last chapter.<a name='more'></a><br />
<br />
<h4>Features</h4>Travis-CI is continuous integration server and nothing else. It builds the project, runs tests and shows tests results in multiple ways. It has some limitations listed in a last sub-chapter, but they should not matter much for most open source projects.<br />
<br />
<h5>Basics</h5>The service is integrated with Github and you must host your project on Github if you want to use it. Once configured, Github sends notifications about every repository change to Travis. <br />
<br />
Sign up and configuration are pretty straightforward and explained in the <a href="http://about.travis-ci.org/docs/user/getting-started/">getting started</a> document.<br />
<br />
Each time there is a change in any branch of the project, Travis-CI will:<br />
<ul><li>initialize a <a href="http://about.travis-ci.org/docs/user/ci-environment/">clean virtual machine</a> with Ubuntu Linux,</li>
<li>clone project repository from Github,</li>
<li>configure the machine,</li>
<li>build changed branch of the project,</li>
<li>run tests,</li>
<li>report test results.</li>
</ul><br />
In addition, Travis-CI automatically merges and tests all incoming pull requests. Tests results of pull requests are integrated into <a href="http://about.travis-ci.org/blog/2012-09-04-pull-requests-just-got-even-more-awesome/">Github GUI</a>. This feature is quite impressive, because you know whether the new code breaks your build without having to merge and test it manually.<br />
<br />
<h5>Test Results</h5>All test results of all public repositories are public. This is probably the main difference against the Travis Pro version. Tests results of private repositories are never published.<br />
<br />
Tests results are available in four different ways:<ul><li>email,</li>
<li>web page,</li>
<li>status image,</li>
<li>json API.</li>
</ul><br />
<h6>Email</h6>Email is sent to the repository owner whenever build fails. It is sent also in case of virtual machine error and when the build starts working again after previous fails.<br />
<br />
<h6>Web Page</h6>Each repository has its own page with list of builds, status of each build and console output of each used virtual machine. You can try less4j <a href="http://travis-ci.org/#!/SomMeri/less4j">test results page</a> to see how it looks like.<br />
<br />
Full console output turned out to be especially useful. Our project behaved differently on Travis-CI environment and we could turn on logging and see everything that was going on. Debugging is much easier if you do not have to guess and can see any information you want.<br />
<br />
<h6>Status Image</h6><a href="http://about.travis-ci.org/docs/user/status-images/">Status image</a> represents last tests result of the project or specified branch. It is green when the build passed, red if it failed and grey if it is still running. You can add it to your project site, readme file or any other web page. <br />
<br />
Current status image of Less4j: <img src="https://secure.travis-ci.org/SomMeri/less4j.png" alt="Less4j Build Status">.<br />
<br />
<h6>Json API</h6>Finally, you can fetch test results through <a href="http://about.travis-ci.org/docs/dev/api/">public API</a>.<br />
<br />
<h5>Limitations</h5>Due to limited computing time, build time of each project is limited to 20 minutes. In addition, tests are not necessary performed immediately. Requests wait in queue until Travis finds free resources to perform them. <br />
<br />
<h4>Under the Hood</h4>This chapter contains information about Travis-CI infrastructure and organization behind it. If you are not interested, <a href="#experiences">skip</a> to the next chapter.<br />
<br />
<h5>Software</h5>The software running on the server was written in ruby and is open source. Its source code is published on <a href="https://github.com/travis-ci">Travis-CI</a> Github account. The core repository has the same name - <a href="https://github.com/travis-ci/travis-ci">Travis-CI</a>. <br />
<br />
It was designed as a distributed testing service for huge number of projects, builds, pull requests, branches and so on. Running it locally for a small number of projects seems like a huge overkill, but you can do it if you want. <br />
<br />
Technical overview is a part of Travis-CI repository <a href="https://github.com/travis-ci/travis-ci#readme">readme</a>. In addition, both local configuration and Travis-CI architecture are explained in this <a href="http://vzmind.tumblr.com/post/9412611799/why-travis">blog post</a>.<br />
<br />
<h5>Hardware</h5>Hosted server runs partly on Heroku and partly on members infrastructure. <br />
<br />
<h5>Organization and Funding</h5>Although Travis-CI documentation does pretty good job with technical information, finding out who they are and how they are funded was a bit harder. This was weird, because hosting and organizing such service must costs a lot of effort and money. <br />
<br />
Anyway, the information is on Travis-CI <a href="https://love.travis-ci.org/">donations page</a>. Scroll down to FAQ on the bottom and all is written there. The project started a German company Travis CI GmbH in Berlin and lives mostly out of donations, founders money and from what they earn on <a href="http://beta.travis-ci.com/">Travis Pro</a>.<br />
<br />
The same page contains also <a href="https://love.travis-ci.org/#future-plans">future plans</a> and does a bit of show about donations. In short, if you donate, they will send you stickers, ringtones, personal thanks and so on. More importantly, you can get also discounts on future Travis-Pro features.<br />
<br />
<a name="experiences"></a><h4>Experiences</h4>We setted up Travis-CI a few days ago, so our experiences are quite short. Travis-CI turned out to be very easy to use. Its documentation is easy to read and console output very practical. Project configuration have been straightforward without any problems. <br />
<br />
Tests have been performed withing few minutes most of the time. However, we have seen also a slow day, when it took few hours to run the test. At one point, Travis-CI started to ignore our project changes. The problem was fixed after we forced build from Github administration page.<br />
<br />
Overall, results are very good. We assume that response to private repositories is going to be a bit faster, simply because they are paid.<br />
</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com4tag:blogger.com,1999:blog-3355333693246614846.post-74802537168011736252012-09-01T01:00:00.002-07:002012-12-16T07:22:04.494-08:00Tackling Comments in ANTLR Compiler<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;"><style type="text/css">
div.blogger-clickTrap { display: none }
</style>Most compilers treat comments as just another meaningless whitespaces. They identify them in the source code and then throw them away. On the other hand, things are a bit harder if you need to know comments content and their position.<br />
<br />
Such requirement is not too common and official ANTLR documentation does not <a href="href="http://www.antlr.org/wiki/pages/viewpage.action?pageId=557063">says much</a> about how to do it. The compiler we have been working on needed to preserve comments, so we had to come up with our own solution. This post documents it.<br />
<br />
First chapter introduces our compiler. It describes what it does, where it is located and how it is written. Second chapter explains our comment preserving requirements. The rest of the post describes datastructures and algorithms used in our solution. <a name='more'></a><br />
<br />
<h4>Table of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><ul><li><a href="#Less4jProject">Less4j Project</a></li>
<ul><li><a href="#Less4jProjectLess">Less</a></li>
<li><a href="#Less4jProjectArchitecture">Architecture</a></li>
<ul><li><a href="#Less4jProjectArchitectureTreeBuilding(Step1)">Tree Building (Step 1)</a></li>
<li><a href="#Less4jProjectArchitectureTheCompiler(Step2)">The Compiler (Step 2)</a></li>
<li><a href="#Less4jProjectArchitecturePrinting(Step3)">Printing (Step 3)</a></li>
</ul></ul><li><a href="#Requirements">Requirements</a></li>
<ul><li><a href="#RequirementsKeepAllComments">Keep All Comments</a></li>
<li><a href="#RequirementsKeepCommentsPositions">Keep Comments Positions</a></li>
<li><a href="#RequirementsExamples">Examples</a></li>
</ul><li><a href="#Datastructures">Datastructures</a></li>
<ul><li><a href="#DatastructuresFirstAbstractSyntaxTree">First Abstract Syntax Tree</a></li>
<li><a href="#DatastructuresFinalAbstractSyntaxTree">Final Abstract Syntax Tree</a></li>
<li><a href="#DatastructuresPrinting">Printing</a></li>
</ul><li><a href="#MatchingCommentsLexer">Matching Comments - Lexer</a></li>
<ul><li><a href="#MatchingCommentsLexerRegularExpression">Regular Expression</a></li>
<li><a href="#MatchingCommentsLexerConsequencesforParser">Consequences for Parser</a></li>
<li><a href="#MatchingCommentsLexerHidingTokens">Hiding Tokens</a></li>
</ul><li><a href="#CollectingHiddenComments">Collecting Hidden Comments</a></li>
<ul><li><a href="#CollectingHiddenCommentsCommonTokenStream">Common Token Stream</a></li>
<li><a href="#CollectingHiddenCommentsCollectingHiddenTokens">Collecting Hidden Tokens</a></li>
</ul><li><a href="#MergingTokensWithAST">Merging Tokens With AST</a></li>
<ul><li><a href="#MergingTokensWithASTCustomizingParser">Customizing Parser</a></li>
<ul><li><a href="#MergingTokensWithASTCustomizingParserCustomTreeNode">Custom Tree Node</a></li>
<li><a href="#MergingTokensWithASTCustomizingParserConfigureParser">Configure Parser</a></li>
</ul><li><a href="#MergingTokensWithASTAssignment">Assignment</a></li>
<ul><li><a href="#MergingTokensWithASTAssignmentPrecedingList">Preceding List</a></li>
<li><a href="#MergingTokensWithASTAssignmentFollowingList">Following List</a></li>
<li><a href="#MergingTokensWithASTAssignmentOrphanList">Orphan List</a></li>
</ul><li><a href="#MergingTokensWithASTAlgorithm">Algorithm</a></li>
<ul><li><a href="#MergingTokensWithASTAlgorithmTheCode">The Code</a></li>
<li><a href="#MergingTokensWithASTAlgorithmExample">Example</a></li>
<li><a href="#MergingTokensWithASTAlgorithmHowItWorks">How It Works</a></li>
<li><a href="#MergingTokensWithASTAlgorithmSpecialCase">Special Case</a></li>
</ul></ul><li><a href="#Consequences">Consequences</a></li>
<li><a href="#FinalNotes">Final Notes</a></li>
</ul></div></div><a name="Less4jProject"></a><h4>Less4j Project</h4><a href="https://github.com/SomMeri/less4j">Less4j</a> is a port of <a href="http://lesscss.org/">Less.js</a> into java. Less.js compiles CSS-like language named less into standard CSS. It is still in prototype phase and compiles subset of CSS back into CSS. Less4j source code is located on Github.<br />
<br />
First sub-chapter introduces less language and is written to be as short as possible. Second one contains overall architecture of our project.<br />
<br />
<a name="Less4jProjectLess"></a><h5>Less</h5>Less is basically CSS with variables, mixins, functions and few other features. Less.js compiles it into regular CSS. The original compiler was written in JavaScript and can be integrated into java project using Rhino.<br />
<br />
Official documentation can be found on <a href="http://lesscss.org/">lesscss.org</a> and introduction tutorial in <a href="http://meri-stuff.blogspot.sk/2012/06/inheritance-and-variables-in-css-with.html">previous post</a> on this blog.<br />
<br />
<a name="Less4jProjectArchitecture"></a><h5>Architecture</h5>Less4j compiler is based on <a href="http://www.antlr.org/">antlr</a> parser generator. This post assumes that you know antlr basics: what is lexer, parser, token and what is a rule. In any case, antlr details are available in our two <a href="http://meri-stuff.blogspot.sk/2011/08/antlr-tutorial-hello-word.html">previous</a> <a href="http://meri-stuff.blogspot.sk/2011/09/antlr-tutorial-expression-language.html">tutorial</a> posts. <br />
<br />
Current version of our project runs in three phases: <ul><li>build abstract syntax tree corresponding to input .less file,</li>
<li>translate .less abstract syntax tree into corresponding .css abstract syntax tree,</li>
<li>create final .css output.</li>
</ul><br />
The rest of this post focus on the abstract syntax tree building e.g, step 1. Second step is not done yet and the third one is more or less trivial. <br />
<br />
<a name="Less4jProjectArchitectureTreeBuilding(Step1)"></a><h6>Tree Building (Step 1)</h6>Abstract syntax tree (AST) is built in two sub-phases:<ul><li>antlr parses input into and creates "first AST",</li>
<li>the "first tree" is transformed into final abstract syntax tree. </li>
</ul><br />
First syntax tree is composed of default antlr tree nodes. They are all instances of <code>CommonTree</code> class and each knows its own type, position in the source code and can return a list of its children. <br />
<br />
Nodes in the final tree belong to <code>ASTCssNode</code> hierarchy. The hierarchy has a separate class for each node type (ruleset, declaration, font-face, ...). Each class has only properties relevant for represented css element. For example, ruleset can return its own selectors and declarations.<br />
<br />
It would be possible to merge these two phases into one. We decided not to do so, because debugging is much easier if things are done in separate phases.<br />
<br />
<a name="Less4jProjectArchitectureTheCompiler(Step2)"></a><h6>The Compiler (Step 2)</h6>The compiler was not yet implemented.<br />
<br />
<a name="Less4jProjectArchitecturePrinting(Step3)"></a><h6>Printing (Step 3)</h6>Printing is essentially breath first traversal through the final abstract syntax tree. The algorithm knows how to print each type of the node and recursively calls itself to print all its childs.<br />
<br />
<a name="Requirements"></a><h4>Requirements</h4>Original less.js preserve comments and their positions. Whitespaces around comments are not preserved, the translation may remove them or add new ones into the output.<br />
<br />
We decided not to mimic the exact less.js behavior. Instead, we tried to identify most important properties of their algorithm and use them as requirements. <br />
<br />
<a name="RequirementsKeepAllComments"></a><h5>Keep All Comments</h5>The first requirement is the most important:<ul><li>No comment may be lost. Every input comment must appear in the output.</li>
</ul><br />
<a name="RequirementsKeepCommentsPositions"></a><h5>Keep Comments Positions</h5>The second most important thing is to keep comments where they have been found. Comments usually describe the element located before them, after them or on the same line. Therefore, comments must not change their location too much during the translation. <br />
<br />
To make things less vague, we defined "keeping the location" as: <ul><li>Comment located between two elements must stay between those two elements.</li>
<li>If the input contains new line between comment and its following element, then its corresponding output comment must be followed by a new line.</li>
<li>If the input line ends with a comment, the translation must keep that comment on the same line as its previous input element.</li>
</ul><br />
All three situations are shown in the next sections input-output examples.<br />
<br />
<a name="RequirementsExamples"></a><h5>Examples</h5>Following two inputs must be unchanged by the translation. In both cases, input and output string must be exactly the same. <br />
<br />
Example 1, ruleset with multiple declarations:<br />
<pre class="brush:css">/*
Preceding comment with new line.
*/
/* before */ .comments /* after */ {
/* Leave me here. */
/* 2 */ color: #808080 /* grey */ , /* blue */ #ffa500;
margin: 2px;/* something about declaration*/
padding: 2px /*before comma*/;/* after comma */
/* Leave me here. */
} /* Almost last comment. */
/* Last comment. */
</pre><br />
Example 2, an empty ruleset:<br />
<pre class="brush:css">/* 1*/ .comments /* 2*/ /* 3 */ .something {
/* Orphan comment. */
}
</pre><br />
<a name="Datastructures"></a><h4>Datastructures</h4>We decided to attach comments and new lines to their surrounding elements. This requires only small changes in our abstract syntax trees and in the printing algorithm.<br />
<br />
Three building and the algorithm for comments attaching are described in following chapters.<br />
<br />
<a name="DatastructuresFirstAbstractSyntaxTree"></a><h5>First Abstract Syntax Tree</h5>Each comment or new line is attached to exactly one tree node. Each node owns comments and new lines attached to it and knows their positions. It knows whether they should go before the node, after the node or inside it. <br />
<br />
Each node in the first abstract syntax tree keeps three lists:<ul><li>list of owned preceding comments and new lines,</li>
<li>list of owned following comments and new lines,</li>
<li>list of owned comments and new lines that are located inside it.</li>
</ul><br />
Each comment and new line is assigned to exactly one of these lists: <pre class="brush:java">private List<Token> preceding = new LinkedList<Token>();
private List<Token> orphans = new LinkedList<Token>();
private List<Token> following = new LinkedList<Token>();
</pre><br />
<a name="DatastructuresFinalAbstractSyntaxTree"></a><h5>Final Abstract Syntax Tree</h5>The final abstract syntax tree keeps comments in three similar lists. The only difference it that each final comment knows whether it should be followed by a new line or not. It does not know how many new lines have originally been there and new lines themselves are thrown away.<br />
<br />
We have two types of comments, those that are followed by new line and those that are not followed by new line:<pre class="brush:java">public class Comment extends ASTCssNode {
private String comment;
private boolean hasNewLine;
//...
public String getComment() {
return comment;
}
public boolean hasNewLine() {
return hasNewLine;
}
}
</pre><br />
Each node in the final abstract syntax tree keeps three lists:<ul><li>list of owned preceding comments,</li>
<li>list of owned following comments,</li>
<li>list of owned comments that are located inside it.</li>
</ul><br />
The class:<pre class="brush:java">class ASTCssNode {
private List<Comment> openingComments = new ArrayList<Comment>();
private List<Comment> orphanComments = new ArrayList<Comment>();
private List<Comment> trailingComments = new ArrayList<Comment>();
// ...
}
</pre><br />
<a name="DatastructuresPrinting"></a><h5>Printing</h5>The printing algorithm requires three changes:<ul><li>It must print all preceding comments before each encountered node.</li>
<li>It must print all following comments after each encountered node.</li>
<li>If it is possible for a node to have orphan comments, then the algorithm must print them too. </li>
</ul><br />
<a name="MatchingCommentsLexer"></a><h4>Matching Comments - Lexer</h4>Both this and the next chapters assume that you already know what is lexer, parser, token, rule and how antlr works. You do not need to know details, everything needed in covered in the antlr <a href="http://meri-stuff.blogspot.sk/2011/08/antlr-tutorial-hello-word.html">hello world</a> tutorial.<br />
<br />
First thing we need to do is to modify the lexer to generate comment tokens. This is covered in the first sub-chapter. Second sub-chapter explains why we can not treat comment tokens the same way as other tokens. Finally, the third sub-chapter shows how to hide them.<br />
<br />
<a name="MatchingCommentsLexerRegularExpression"></a><h5>Regular Expression</h5>CSS comments look a lot like C or Java comments. They start with <code>/*</code> symbol, end with <code>*/</code> symbol and can not be nested. Therefore, we have to match anything in between <code>/*</code> and its closest <code>*/</code>:<br />
<pre class="brush:java">COMMENT: '/*' ( options { greedy=false; } : .*) '*/';
</pre><br />
The <code>options { greedy=false; }</code> part is necessary, because antlr lexer is greedy. If we would use a simple regular expression <code>'/*' .* '*/'</code>, the lexer would create one big token from everything between the first <code>/*</code> and the last <code>*/</code> in the input. <br />
<br />
<a name="MatchingCommentsLexerConsequencesforParser"></a><h5>Consequences for Parser</h5>Comments and whitespaces have one thing in common: they can appear in any number between any other two tokens. If we would use the above rule as is, then we would have to add comments tokens into every possible place in each parser rule. <br />
<br />
For example, following declaration rule:<br />
<pre class="brush:java">declaration: property COLON expr prio?;
</pre>would have to be changed into something like this: <br />
<pre class="brush:java">declaration: property COMMENT* COLON COMMENT* expr COMMENT* prio?;
</pre><br />
This way of comments handling is very error prone, labor extensive and we doubt anybody ever handled them this way. We do not want to add <code>COMMENT*</code> into every possible place in our grammar, but we need them in final abstract syntax tree.<br />
<br />
<a name="MatchingCommentsLexerHidingTokens"></a><h5>Hiding Tokens</h5>Each generated token has property named "channel". Only tokens with channel value 0 reach the parser, all other channels are hidden. It is important to remember that lexer generates token stream with all tokens including the hidden ones. Hidden tokens are filtered out between the parser and the lexer.<br />
<br />
If you want to hide a token, you can use either a non-zero channel number or the <code>HIDDEN</code> keyword. The keyword corresponds to channel number 99. <br />
<br />
Use the action part of the lexer rule to send all comments on hidden channel:<br />
<pre class="brush:java">COMMENT: '/*' ( options { greedy=false; } : .*) '*/'
{
$channel = HIDDEN;
};
</pre><br />
<a name="CollectingHiddenComments"></a><h4>Collecting Hidden Comments</h4>We have successfully added comments into the grammar without having to rewrite every single rule in it. Of course, as comments disappeared from the parser, we have to collect them elsewhere and then attach them to the abstract syntax tree.<br />
<br />
<a name="CollectingHiddenCommentsCommonTokenStream"></a><h5>Common Token Stream</h5>Antlr generated lexer and parser are not compatible with each other. While the lexer implements the <code>TokenSource</code> interface, the parser takes an instance of <code>TokenStream</code> as its input. This class level incompatibly makes sense because:<ul><li>lexer generates hidden tokens that should be filtered out,</li>
<li>parser occasionally needs to look ahead into upcoming tokens, but lexer does not have the ability to show them. Lexer returns one token at a time and is not able to show following or previous tokens.</li>
</ul><br />
Typical way to connect lexer with parser is to use <code>CommonTokenStream</code>. Common token stream adds look ahead abilities to the token source and filters out unwanted tokens:<br />
<pre class="brush:java">LessLexer lexer = new LessLexer(inputStream);
CommonTokenStream tokens = new CommonTokenStream(lexer);
LessParser parser = new LessParser(tokens);
</pre><br />
<a name="CollectingHiddenCommentsCollectingHiddenTokens"></a><h5>Collecting Hidden Tokens</h5>The easiest way to collect hidden tokens is to place collector between lexer and common token stream. The collector implements the same interface as the lexer (e.g. <code>TokenSource</code>), wraps over the lexer and acts as its replacement.<br />
<br />
Common token source is given a collector instead of the lexer as its input:<br />
</div><pre class="brush:java">LessLexer lexer = new LessLexer(inputStream);
CollectorTokenSource tokenSource = new CollectorTokenSource(lexer, ...);
CommonTokenStream tokens = new CommonTokenStream(tokenSource);
LessParser parser = new LessParser(tokens);
</pre><div style="text-align: justify;"><br />
Common token stream uses <code>Token nextToken()</code> method of the <code>TokenSource</code> interface to read tokens. Our collector delegates the call to the lexer and returns the same token as the lexer would. However, if it is either a comment or a new line, the collector will store the token in a list of collected tokens:</div><pre class="brush:java">class CollectorTokenSource implements TokenSource {
...
@Override
public Token nextToken() {
Token nextToken = source.nextToken();
if (shouldCollect(nextToken)) {
collectedTokens.add(nextToken);
}
return nextToken;
}
protected boolean shouldCollect(Token nextToken) {
// filter the token by its type
return collectTokenTypes.contains(nextToken.getType());
}
}
</pre><div style="text-align: justify;"><br />
Full implementation of the <code>CollectorTokenSource</code> class:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">class CollectorTokenSource implements TokenSource {
private final TokenSource source;
private final Set<Integer> collectTokenTypes =
new HashSet<Integer>();
private final LinkedList<Token> collectedTokens =
new LinkedList<Token>();
public CollectorTokenSource(TokenSource source,
Collection<Integer> collectTokenTypes) {
super();
this.source = source;
this.collectTokenTypes.addAll(collectTokenTypes);
}
/**
* Returns next token from the wrapped token source. Stores it
* in a list if necessary.
*/
@Override
public Token nextToken() {
Token nextToken = source.nextToken();
if (shouldCollect(nextToken)) {
collectedTokens.add(nextToken);
}
return nextToken;
}
/**
* Decide whether collect the token or not.
*/
protected boolean shouldCollect(Token nextToken) {
// filter the token by its type
return collectTokenTypes.contains(nextToken.getType());
}
public LinkedList<Token> getCollectedTokens() {
return collectedTokens;
}
@Override
public String getSourceName() {
return "Collect hidden channel " + source.getSourceName();
}
}</pre></div><br />
</div></div><a name="MergingTokensWithAST"></a><h4>Merging Tokens With AST</h4>Collected tokens need to be merged with abstract syntax tree generated by antlr. To do so, we have to customize the tree generated by antlr parser and then attach hidden tokens to it. <br />
<br />
Merge is possible because: <ul><li>Each antlr token knows his own index in the token stream.</li>
<li>Each antlr node knows indexes of both the first and last tokens that belongs to it.</li>
</ul><br />
First sub-chapter shows how to customize the tree and the rest of the chapter explains how to merge it with collected tokens.<br />
<br />
<a name="MergingTokensWithASTCustomizingParser"></a><h5>Customizing Parser</h5>Tree nodes generated by antlr parser are unable to hold any additional information. As we want to attach hidden tokens directly to the tree, we have to create new type of tree node and configure generated parser to use it. <br />
<br />
<a name="MergingTokensWithASTCustomizingParserCustomTreeNode"></a><h6>Custom Tree Node</h6>Abstract syntax tree generated by default is composed of instances of <code>CommonTree</code> class. <br />
<br />
We extended <code>CommonTree</code> and added three properties into new class:<pre class="brush:java">public class HiddenTokenAwareTree extends CommonTree {
private List<Token> preceding = new LinkedList<Token>();
private List<Token> orphans = new LinkedList<Token>();
private List<Token> following = new LinkedList<Token>();
// ... constructors, getters and setters follow
}</pre><br />
<a name="MergingTokensWithASTCustomizingParserConfigureParser"></a><h6>Configure Parser</h6>Generated parser delegates all tree modifying operations to <code>TreeAdaptor</code> interface. Tree adaptor creates new nodes, add childs to them or deletes them when needed. Default tree adapter is called <code>CommonTreeAdaptor</code>.<br />
<br />
As our tree nodes extend default <code>CommonTree</code> class, our custom adaptor needs to customize only the <code>create</code> method. Everything else may work the same way as in the original <code>CommonTreeAdaptor</code>:<br />
<pre class="brush:java">class LessTreeAdaptor extends CommonTreeAdaptor {
@Override
public Object create(Token payload) {
return new HiddenTokenAwareTree(payload);
}
}
</pre><br />
Tree adaptor used by parser is configured via <code>setTreeAdaptor</code> method:<br />
<pre class="brush:java">LessParser parser = new LessParser(tokens);
parser.setTreeAdaptor(new LessTreeAdaptor());</pre><br />
<a name="MergingTokensWithASTAssignment"></a><h5>Assignment</h5>Our algorithm is supposed to assign each collected token to either preceding, following or orphan list of some tree node. <br />
<br />
A tree node may own only those hidden tokens that are directly before it, directly after it or directly inside it. "Directly" means that no other tree node starts or ends between the hidden token and its owning node. There must be no other closer candidate.<br />
<br />
<a name="MergingTokensWithASTAssignmentPrecedingList"></a><h6>Preceding List</h6>The list of preceding tokens contains hidden tokens located directly before the owning node. <br />
<br />
For example, this ruleset:<pre class="brush:css">/* 0 */.comments /* 1 */ /* 2 */ .something /* 3 */ {
declaration: value
}
</pre><br />
corresponds to following simplified abstract syntax tree (the parenthesis contains start and stop token indexes of the subtree):<br />
<pre class="brush:text">RULESET (2-22)
- SELECTOR (2-10)
--- CSS_CLASS comments (2-3)
--- EMPTY_COMBINATOR (9-9)
--- CSS_CLASS something (9-11)
- BODY_OF_DECLARATIONS (14-22)
--- DECLARATION (17-20)
----- PROPERTY declaration (17-17)
----- TERM value (20-20)
</pre><br />
there are four hidden comments and one hidden new line (the parenthesis contains token index):<pre class="brush:text">(0) COMMENT /* 0 */
(5) COMMENT /* 1 */
(7) COMMENT /* 2 */
(12) COMMENT /* 3 */
(15) NEW_LINE
</pre><br />
The comment 0 goes to the <code>RULESET</code>, because there is nothing that would start closer to it. Although <code>SELECTOR</code> node starts with the same index as the ruleset, it is rulesets child and thus considered to be behind it.<br />
<br />
Comments 1 and 2 go to the <code>EMPTY_COMBINATOR</code> and the comment 3 and new line belong to the <code>BODY_OF_DECLARATIONS</code>.<br />
<br />
<a name="MergingTokensWithASTAssignmentFollowingList"></a><h6>Following List</h6>The "following" list is going to be used only if there is no possible preceding owner. It may contain only hidden tokens located directly after the owning node. <br />
<br />
For example, both comments in the following example have no element directly behind them:<pre class="brush:css">.comments .something {
declaration: something;
/* Trailing comment. */
}
.other .selector {
declaration: value;
}
/* Last comment. */
</pre><br />
The input corresponds to following simplified tree (the parenthesis contains start and stop token indexes of the subtree):<pre class="brush:text">RULESET (0-18)
- SELECTOR .comments .something (0-4)
- BODY_OF_DECLARATIONS (6-18)
--- DECLARATION declaration: something; (9-13)
RULESET (22-37)
- SELECTOR .other .selector (22-26)
- BODY_OF_DECLARATIONS (28-37)
--- DECLARATION declaration: value; (31-35)
</pre><br />
hidden channel contains two comments (we omitted new lines because there are too many of them):<pre class="brush:text">(16) COMMENT /* Trailing comment. */
(39) COMMENT /* Last comment. */
</pre><br />
The <code>Trailing comment.</code> has index 16. There is nothing between this comment and the end of the first ruleset (position 18). No further element can own the comment, because there must be no node start nor node end between the hidden token and its owning node.<br />
<br />
Therefore we have to look for an element that ends directly before it. There is nothing between the end of <code>declaration: something;</code> and the comment on position 16. The comment should be assigned to that declaration as the following hidden token.<br />
<br />
The second comment <code>Last comment.</code> is the last comment in the input file. It will be assigned to the last element in the file e.g., second <code>RULESET</code>.<br />
<br />
<a name="MergingTokensWithASTAssignmentOrphanList"></a><h6>Orphan List</h6>The list of orphan comments should contain only those comments that have neither preceding nor following elements:<br />
<pre class="brush:css">.comments .something {
/* Orphan comment. */
}
</pre><br />
The only comment in the above example belongs to the ruleset body.<br />
<br />
<a name="MergingTokensWithASTAlgorithm"></a><h5>Algorithm</h5>The solution is implemented in class named <code>ListToTreeCombiner</code>. The list of hidden tokens is stored in its private property, so we do not have to constantly pass it around.<br />
<br />
Root of the abstract syntax tree is special case and we will ignore it for now. The assignment of hidden tokens to any other tree node is done inside the <code>associateAsChild</code> method. It takes tree node as a parameter and assigns all hidden tokens up to its end. The method:<ul><li>reads all hidden tokens up to the end of the input sub-tree,</li>
<li>assigns them to the input node or one of its childs,</li>
<li>removes all used tokens from the list of hidden tokens.</li>
</ul><br />
The method works correctly only if the list of hidden tokens starts with tokens that should be assigned to the sub-tree in parameter. <br />
<br />
This part explains how the <code>associateAsChild</code> method works. First section shows both commented code of this method and full listing of <code>ListToTreeCombiner</code> class. Second section contains the input we will use as the example. Remaining sections apply the method to the example input and step through it to show how it works.<br />
<br />
<a name="MergingTokensWithASTAlgorithmTheCode"></a><h6>The Code</h6>Next code example contains the <code>associateAsChild</code> method. The full class including all helper methods is shown below:<br />
<pre class="brush:java">private LinkedList<Token> hiddenTokens;
private void associateAsChild(HiddenTokenAwareTree ast) {
//fill preceding list of ast
addAllPrecedingTokens(ast);
LinkedList<HiddenTokenAwareTree> children = getChildren(ast);
//if the node has no children, fill orphans list
if (children.isEmpty()) {
addAllContainedTokens(ast);
return;
}
HiddenTokenAwareTree previousChild = null;
for (HiddenTokenAwareTree child : children) {
//everything up to first new line or next child belongs to
//previous element
assignFirstCommentsSegment(previousChild, child);
//recursion assigns tokens to the child
associateAsChild(child);
previousChild = child;
}
//remaining tokens located inside this node belong to the last child
addFollowingTokens(previousChild, ast.getTokenStopIndex());
}
</pre><br />
The whole class:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">public class ListToTreeCombiner {
private LinkedList<Token> hiddenTokens;
public void associate(HiddenTokenAwareTree ast,
LinkedList<Token> hiddenTokens) {
initialize(hiddenTokens);
LinkedList<HiddenTokenAwareTree> children = getChildren(ast);
for (HiddenTokenAwareTree child : children) {
associateAsChild(child);
}
if (children.isEmpty()) {
addAllContainedTokens(ast);
} else {
HiddenTokenAwareTree lastChild = children.getLast();
lastChild.addFollowing(hiddenTokens);
}
}
private void initialize(LinkedList<Token> hiddenTokens) {
this.hiddenTokens = hiddenTokens;
}
private void associateAsChild(HiddenTokenAwareTree ast) {
addAllPrecedingTokens(ast);
LinkedList<HiddenTokenAwareTree> children = getChildren(ast);
if (children.isEmpty()) {
addAllContainedTokens(ast);
return;
}
HiddenTokenAwareTree previousChild = null;
for (HiddenTokenAwareTree child : children) {
assignFirstCommentsSegment(previousChild, child);
associateAsChild(child);
previousChild = child;
}
addFollowingTokens(previousChild, ast.getTokenStopIndex());
}
private void assignFirstCommentsSegment(
HiddenTokenAwareTree firstChild,
HiddenTokenAwareTree secondChild) {
LinkedList<Token> tail =
readTillNewLine(secondChild.getTokenStartIndex());
if (!tail.isEmpty() && firstChild != null) {
if (tail.peekLast().getType() == LessLexer.NEW_LINE)
firstChild.addFollowing(tail);
else
secondChild.addPreceding(tail);
}
}
private void addAllContainedTokens(HiddenTokenAwareTree ast) {
int stop = ast.getTokenStopIndex();
List<Token> result = readPrefix(stop);
ast.addOrphans(result);
}
private void addFollowingTokens(HiddenTokenAwareTree target,
int stop) {
List<Token> result = readPrefix(stop);
target.addFollowing(result);
}
private LinkedList<HiddenTokenAwareTree> getChildren(
HiddenTokenAwareTree ast) {
List<HiddenTokenAwareTree> children = ast.getChildren();
if (children == null)
return new LinkedList<HiddenTokenAwareTree>();
LinkedList<HiddenTokenAwareTree> copy =
new LinkedList<HiddenTokenAwareTree>(children);
Collections.sort(copy, new PositionComparator());
return copy;
}
private void addAllPrecedingTokens(HiddenTokenAwareTree target) {
int start = target.getTokenStartIndex();
List<Token> tokens = readPrefix(start);
target.addPreceding(tokens);
}
private LinkedList<Token> readTillNewLine(int end) {
LinkedList<Token> result = new LinkedList<Token>();
if (hiddenTokens.isEmpty())
return result;
Token first = hiddenTokens.peekFirst();
while (first != null && first.getTokenIndex() < end &&
first.getType() == LessLexer.COMMENT) {
result.add(first);
hiddenTokens.removeFirst();
first = hiddenTokens.peekFirst();
}
if (first == null || first.getTokenIndex() >= end)
return result;
result.add(first);
return result;
}
private List<Token> readPrefix(int end) {
List<Token> result = new ArrayList<Token>();
if (hiddenTokens.isEmpty())
return result;
Token first = hiddenTokens.peekFirst();
while (first != null && first.getTokenIndex() < end) {
result.add(first);
hiddenTokens.removeFirst();
first = hiddenTokens.peekFirst();
}
return result;
}
class PositionComparator implements Comparator<CommonTree> {
@Override
public int compare(CommonTree arg0, CommonTree arg1) {
return arg0.getTokenStartIndex() - arg1.getTokenStartIndex();
}
}
}</pre></div><br />
</div></div><a name="MergingTokensWithASTAlgorithmExample"></a><h6>Example</h6>We will use the first .less sample from <a href="#Requirements">requirements chapter </a>to show how the method works.<br />
<br />
The example input:<br />
<pre class="brush:css">/*
Preceding comment with new line.
*/
/* before */ .comments /* after */ {
/* Leave me here. */
/* 2 */ color: #808080 /* grey */ , /* blue */ #ffa500;
margin: 2px;/* something about declaration*/
padding: 2px /*before comma*/;/* after comma */
/* Leave me here. */
} /* Almost last comment. */
/* Last comment. */
</pre><br />
Expand to see all collected hidden tokens: <div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;">The number in parenthesis is token index. It is followed by token type and a text corresponding to the token. <br />
<br />
<div style="text-align: left;"><pre class="brush:text">(0) COMMENT /* Preceding comment with new line.*/
(1) NEW_LINE
(2) COMMENT /* before */
(7) COMMENT /* after */
(10) NEW_LINE
(12) COMMENT /* 1 */
(13) NEW_LINE
(15) COMMENT /* 2 */
(22) COMMENT /* grey */
(26) COMMENT /* blue */
(30) NEW_LINE
(36) COMMENT /*before comma*/
(38) COMMENT /* something about declaration*/
(39) NEW_LINE
(45) COMMENT /*before comma*/
(47) COMMENT /* after comma */
(48) NEW_LINE
(50) COMMENT /* Leave me here. */
(51) NEW_LINE
(54) COMMENT /* Almost last comment. */
(55) NEW_LINE
(56) COMMENT /* Last comment. */
</pre></div><br />
</div></div>Simplified abstract syntax tree:<br />
<pre class="brush:text">STYLE_SHEET 4 58
-- RULESET 4 52
---- SELECTOR 4 5
-------- ...
---- BODY_OF_DECLARATIONS 9 52
------ DECLARATION 17 29
-------- ...
------ DECLARATION 32 37
-------- ...
------ DECLARATION 41 46
-------- ...
</pre><br />
<a name="MergingTokensWithASTAlgorithmHowItWorks"></a><h6>How It Works</h6>Assume that we have a sub-tree corresponding to the "body of declarations" in the above example and a list of hidden tokens. All hidden tokes that belong to selector or other previous element have already been assigned and removed from the list. <br />
<br />
As the body of declarations starts with token number 9, any token with index lower than 9 can be assigned as preceding token to the body of declarations. We created method <code>addAllPrecedingTokens(ast)</code> to do that. It takes tree node as a parameter and puts all hidden tokens with low indexes into its preceding list.<br />
<br />
Associate preceding tokens first:<br />
<pre class="brush:java">private void associateAsChild(HiddenTokenAwareTree ast) {
addAllPrecedingTokens(ast);
// ...
}</pre><br />
Expand to see the body of <code>addAllPrecedingTokens</code> helper method:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">private void addAllPrecedingTokens(HiddenTokenAwareTree target) {
int start = target.getTokenStartIndex();
List<Token> tokens = readPrefix(start);
target.addPreceding(tokens);
}
</pre><br />
</div></div>Tokens between 9 and 52 belong to individual declarations inside the body of declarations. As the first child ends up at the position 29, everything up to that position belongs to it. As we already chopped off everything up token 9, we can recursively call our method to do that work. <pre class="brush:java">associateAsChild(child);
</pre><br />
The recursive call assigns everything up to the position 29 to the first child. However, that child may own also tokens with positions higher than 29, but only up to closest new line and only up to the next element start or end. <br />
<br />
Otherwise said, if there is no new line between two childs, all hidden tokens up to the beginning of the second child belong to that child. If there is new line between them, everything up to that new line belongs to the first child.<br />
<br />
This logic is implemented inside the <code>assignFirstCommentsSegment</code> method. It takes two childs as a parameter and decides whether first child should have any following tokens assigned.<br />
<br />
Assigning following tokens is the last thing we had to do with the first child. Once it is done, we can move to the next childs:<pre class="brush:java">LinkedList<HiddenTokenAwareTree> children = getChildren(ast);
HiddenTokenAwareTree previousChild = null;
for (HiddenTokenAwareTree child : children) {
assignFirstCommentsSegment(previousChild, child);
associateAsChild(child);
previousChild = child;
}</pre><br />
Expand to see the <code>assignFirstCommentsSegment</code> methods body:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<div style="text-align: left;"><pre class="brush:java">private void assignFirstCommentsSegment(
HiddenTokenAwareTree firstChild,
HiddenTokenAwareTree secondChild) {
LinkedList<Token> tail =
readTillNewLine(secondChild.getTokenStartIndex());
if (!tail.isEmpty() && firstChild != null) {
if (tail.peekLast().getType() == LessLexer.NEW_LINE)
firstChild.addFollowing(tail);
else
secondChild.addPreceding(tail);
}
}
</pre></div><br />
</div></div>The only special case is the last child which will get also all remaining hidden tokens up body of declaration end at the index 52.<br />
<br />
We use the <code>addFollowingTokens</code> method to assign all remaining tokens to the last child. Click to see its body:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">private void addFollowingTokens(HiddenTokenAwareTree target,
int stop) {
List<Token> result = readPrefix(stop);
target.addFollowing(result);
}
</pre><br />
</div></div>The method may exit now, everything up to the end of its input node has already been assigned.<br />
<pre class="brush:java"> addFollowingTokens(previousChild, ast.getTokenStopIndex());
}
</pre><br />
<a name="MergingTokensWithASTAlgorithmSpecialCase"></a><h6>Special Case</h6>The above example omitted one special case: sub-tree with no childs must associate all hidden comments as its orphan comments.<br />
<br />
<pre class="brush:java">LinkedList<HiddenTokenAwareTree> children = getChildren(ast);
if (children.isEmpty()) {
addAllContainedTokens(ast);
return;
}
</pre><br />
The body of the <code>addAllContainedTokens</code> method:<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">private void addAllContainedTokens(HiddenTokenAwareTree ast) {
int stop = ast.getTokenStopIndex();
List<Token> result = readPrefix(stop);
ast.addOrphans(result);
}
</pre><br />
</div></div><a name="Consequences"></a><h4>Consequences</h4>The approach we just described has few important consequences for our grammar:<ol><li>All parentheses, curly braces and other "structure only" tokens must be present in the abstract syntax tree. We can not suppress them in the grammar.</li>
<li>Major modifications of abstract syntax tree can not be done in the grammar. They can be done only after hidden tokens have been merged with the tree.</li>
<li>We must be careful not to loose comments during the translation of the first abstract syntax tree into the final one. The mapping between the first and final abstract syntax trees is not one-to-one. All comments must find their way into the final tree, even if the owning node is thrown away.</li>
<li>Abstract syntax tree tend to be more complicated then it would be otherwise. Every simple small thing must be a separate node/class. This the most annoying point in this list.</li>
</ol><br />
<a name="FinalNotes"></a><h4>Final Notes</h4>Our project uses multiple phases to create ast:<ul><li>Create antlr abstract syntax tree and collect hidden comments and new lines.</li>
<li>Merge antlr abstract syntax tree with hidden tokens.</li>
<li>Convert antlr abstract syntax tree into final abstract syntax tree.</li>
</ul><br />
It would be possible to merge these three phases into one. Antlr is able to spit out parser that does all these thing automatically and returns abstract syntax tree as we want it. <br />
<br />
We decided not to do it, because separate phases are easier to understand and debug. If it turns out that such approach has significant drawbacks, we will refactor the code and merge phases. Writing second compiler version or refactoring the code is usually easier than writing the first version, especially if the unit tests are already written.</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>
Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com38tag:blogger.com,1999:blog-3355333693246614846.post-88788376438833468772012-08-01T03:55:00.000-07:002013-03-31T01:26:06.949-07:00Wro4j, Page Load Optimization and Less.js<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><style type="text/css">div.blogger-clickTrap {display: none;}</style><div style="text-align: justify;"><a href="http://alexo.github.com/wro4j/">Wro4j</a> is primary a JavaScript and CSS merge and minimization library. Its original purpose was to speed up the page load. However, its final design made it easy to add integration with LESS, CoffeScript and few other technologies.<br />
<br />
Less was introduced in <a href="http://meri-stuff.blogspot.sk/2012/06/inheritance-and-variables-in-css-with.html">previous article</a>. In short, it is CSS with object oriented inheritance, variables and few other additional features. It is compiled into regular style sheets and served to the browser. Less was written in JavaScript and usually runs in the browser. If you want to run it on the server side, you have to use wro4j or some other <a href="meri-stuff.blogspot.sk/2012/06/inheritance-and-variables-in-css-with.html#ServerSideIntegration">integration library</a>.<br />
<br />
The post is mostly <a href="http://alexo.github.com/wro4j/">wro4j</a> tutorial with focus on Less integration. It explains how wro4j works and how to configure it. There is very little about Less and almost everything is about wro4j. <a name='more'></a><br />
<br />
<a name="TOC"></a><h4>Table Of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the table of contents:</a><br />
<div class="answertext" style="text-align: justify;"><ul><li><a href="#Overview">Overview</a></li>
<ul><li><a href="#OverviewLicensingandCode">Licensing and Code</a></li>
<li><a href="#OverviewWhatItDoes">What It Does</a></li>
<li><a href="#OverviewOptimizations">Optimizations</a></li>
<ul><li><a href="#OverviewOptimizationsGrouping">Grouping</a></li>
<li><a href="#OverviewOptimizationsMinimization">Minimization</a></li>
<li><a href="#OverviewOptimizationsEntityTag">Entity Tag</a></li>
<li><a href="#OverviewOptimizationsNoteonEffectivity">Note on Effectivity</a></li>
</ul><li><a href="#OverviewLess.jsWithWro4j">Less.js With Wro4j</a></li>
<ul><li><a href="#OverviewLess.jsWithWro4jWithinFramework">Within Framework</a></li>
<li><a href="#OverviewLess.jsWithWro4jCompileTime">Compile Time</a></li>
<li><a href="#OverviewLess.jsWithWro4jOntheFly">On the Fly</a></li>
</ul></ul><li><a href="#HowitWorks">How it Works</a></li>
<ul><li><a href="#HowitWorksWroFilter">Wro Filter</a></li>
<li><a href="#HowitWorksWroManager">Wro Manager</a></li>
<li><a href="#HowitWorksInjector">Injector</a></li>
<li><a href="#HowitWorksWroManagerFactory">Wro Manager Factory</a></li>
</ul><li><a href="#Configuration">Configuration</a></li>
<ul><li><a href="#ConfigurationMavenDependencies">Maven Dependencies</a></li>
<li><a href="#ConfigurationFilter">Filter</a></li>
<li><a href="#ConfigurationConfiguringGroups">Configuring Groups</a></li>
<ul><li><a href="#ConfigurationConfiguringGroupsConfigurationFileStructure">Configuration File Structure</a></li>
<li><a href="#ConfigurationConfiguringGroupsJsandCssTags">Js and Css Tags</a></li>
<li><a href="#ConfigurationConfiguringGroupsGroupReferences">Group References</a></li>
</ul><li><a href="#ConfigurationUsingGroups">Using Groups</a></li>
<li><a href="#ConfigurationProcessors">Processors</a></li>
<ul><li><a href="#ConfigurationProcessorsReplaceManagerFactory">Replace Manager Factory</a></li>
<li><a href="#ConfigurationProcessorsConfigureManagerFactory">Configure Manager Factory</a></li>
<li><a href="#ConfigurationProcessorsRelativeReferencesinCSS">Relative References in CSS</a></li>
<li><a href="#ConfigurationProcessorsMissing;in.jsFiles">Missing ; in .js Files</a></li>
<li><a href="#ConfigurationProcessorsMinimization">Minimization</a></li>
<li><a href="#ConfigurationProcessorsLessCompiler">Less Compiler</a></li>
</ul></ul><li><a href="#DynamicModel">Dynamic Model</a></li>
<ul><li><a href="#DynamicModelSampleProject">Sample Project</a></li>
<li><a href="#DynamicModelFilter">Filter</a></li>
<li><a href="#DynamicModelTheModel">The Model</a></li>
<ul><li><a href="#DynamicModelTheModelInitialModel">Initial Model</a></li>
<li><a href="#DynamicModelTheModelModelTransformers">Model Transformers</a></li>
</ul><li><a href="#DynamicModelGroupExtractor">Group Extractor</a></li>
<li><a href="#DynamicModelWroManagerFactory">Wro Manager Factory</a></li>
</ul><li><a href="#End">End</a></li>
</ul></div></div><a name="Overview"></a><h4>Overview</h4>We start with general overview of wro4j and its functionality. This chapter does not go into details, it shows only what is possible. <br />
<br />
Licensing information is in the first chapter and the second one contains general overview of what wro4j does. Following sub-chapter lists all available optimizations. Lastly, there are multiple ways how to use wro4j with less.js. All are shown in the last sub-chapter. <br />
<br />
<a name="OverviewLicensingandCode"></a><h5>Licensing and Code</h5>Wro4j is distributed under Apache License 2.0 and can be found either <a href="http://alexo.github.com/wro4j/">on Github</a> or on <a href="http://code.google.com/p/wro4j/">Google Code</a>. Its <a href="https://github.com/alexo/wro4j">source code</a> is hosted on Github.<br />
<br />
Wro4j has nice and readable source code. If you need something it does not provide, it should be easy to patch it. The project is also very responsive to bug fixes. We found one small bug while writing this article and they fixed it within a day. <br />
<br />
<a name="OverviewWhatItDoes"></a><h5>What It Does</h5>Wro4j creates groups of JavaScript and CSS resources. All files within one group are merged together and served to the browser as one file. <br />
<br />
When the browser requests group for the first time, wro4j:<ul><li>applies pre-processors to each file that belongs to the group,</li>
<li>merges all files together,</li>
<li>applies post-processors to the merged group,</li>
<li>places the end result into the cache,</li>
<li>sends the group to the browser.</li>
</ul><br />
Following requests for the same group retrieve the group from the cache. Wro4j will not merge the same group twice.<br />
<br />
Some pre/post-processors are necessary and merged group does not work correctly without them. Others are optional and perform additional optimization or other useful work. Minimizing library is one such processor, less compiler another. <a href="http://code.google.com/p/wro4j/wiki/AvailableProcessors">Complete list</a> of available processors in available on wro4j wiki page.<br />
<br />
<a name="OverviewOptimizations"></a><h5>Optimizations</h5>Wro4j offers three basic optimizations:<ul><li>grouping,</li>
<li>minimization,</li>
<li>entity tag.</li>
</ul><br />
First three sections of this sub-chapter describe these three optimizations and the last one contains short note on their effectivity.<br />
<br />
<a name="OverviewOptimizationsGrouping"></a><h6>Grouping</h6>Grouping lowers number of requests sent to server. If the web page loads each JavaScript or CSS file separately, the server has to handle one request for each resource. If those files are grouped together, then the server has to handle only one request.<br />
<br />
<a name="OverviewOptimizationsMinimization"></a><h6>Minimization</h6>Minimized style sheets and JavaScript files have smaller download size. Shorter resources are loaded faster. <br />
<br />
Minimization strips unnecessary whitespaces and comments. It also shortens long variable and function names.<br />
<br />
<a name="OverviewOptimizationsEntityTag"></a><h6>Entity Tag</h6>Wro4j adds entity tag (<a href="www.freesoft.org/CIE/RFC/2068/187.htm">Etag</a>) code to headers. Entity tag is computed by the server and sent to browser together with the requested resource. It uniquely identifies the resource and its version. <br />
<br />
When the resource content changes, entity tag must change too. Most application use hash of the resource content as etag code.<br />
<br />
When the browser needs the same file second time, it adds etag code into the request. The server then compares incoming etag code with etag of current resource version. If they match, the server sends back "304 (Not Modified)" status to the browser.<br />
<br />
Otherwise said, whenever the resource changes on the server, the browser will download its new version. If the resource was unchanged, the browser will use cached version. <br />
<br />
The full description of the protocol is <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">available here</a>.<br />
<br />
<a name="OverviewOptimizationsNoteonEffectivity"></a><h6>Note on Effectivity</h6>We do not know how much each of these optimizations help. We could not find an article or post that would measure how much these things help and we had not done our own experiments.<br />
<br />
In any case, minimization and grouping are generally recommended as a best practice.<br />
<br />
<a name="OverviewLess.jsWithWro4j"></a><h5>Less.js With Wro4j</h5>There are three different ways how to use less.js with wro4j:<ul><li>withing framework,</li>
<li>compile time,</li>
<li>on the fly.</li>
</ul><br />
<a name="OverviewLess.jsWithWro4jWithinFramework"></a><h6>Within Framework</h6>Add wro4j and configure it to use less compiler as one of processors. This option requires the most configuration, but you will be able to use all features and functionality that wro4j offers. <br />
<br />
The rest of this post explains how to do it. <br />
<br />
<a name="OverviewLess.jsWithWro4jCompileTime"></a><h6>Compile Time</h6>Use Maven and wro4j to compile less.js into css at compile time. <br />
<br />
A simple <a href="http://blog.frankel.ch/empower-your-css-in-your-maven-build">step by step tutorial</a> on how to do it is on Nicolas Fränkels blog. In addition to what is written there, Eclipse users will have to do some additional steps described in <a href="https://community.jboss.org/en/tools/blog/2012/01/17/css-and-js-minification-using-eclipse-maven-and-wro4j">JBoss Tools article</a>.<br />
<br />
The detailed official documentation is on projects <a href="http://code.google.com/p/wro4j/wiki/MavenPlugin">google code page</a>.<br />
<br />
<a name="OverviewLess.jsWithWro4jOntheFly"></a><h6>On the Fly</h6>If you do not care about optimizations and caching, you can use wro4j to compile .less files <a href="http://code.google.com/p/wro4j/wiki/AbstractProcessorsFilter">on the fly</a>. <br />
<br />
Such configuration is simpler than what we will show in following chapters, but has also some disadvantages. The biggest one is that the compiler only variant does not support caching. Each .less file is compiled each time it is accessed. <br />
<br />
<a name="HowitWorks"></a><h4>How it Works</h4>The final thing to explain before starting with the configuration is how wro4j works inside. It is not necessary to read this chapter in order to understand the rest of the post. <br />
<br />
If you are not interested in wro4j internals, skip to the <a href="#Configuration">next chapter</a>.<br />
<br />
<a name="HowitWorksWroFilter"></a><h5>Wro Filter</h5>Entry point into wro4j is servlet filter. It listens to browser requests and initiates all wro4j work when those requests come. Almost no work is done in the filter, everything is delegated to other classes. <br />
<br />
If no group corresponds to uri in request, filter logs an error and forwards the request to other servlet filters. <br />
<br />
<a name="HowitWorksWroManager"></a><h5>Wro Manager</h5>Wro manager is the class responsible for all wro4j work. It manages caching, resource locators, applies pre-processors to individual files, merges them, applies post-processors to the result and so on. <br />
<br />
It delegates a lot to other classes and requires extensive configuration. It must be supplied several strategy, factory and helper objects. Additionally, wro4j has its own inversion of control framework and wro manager must both configure injection system and be injected by it. <br />
<br />
<code>WroManager</code> is very locked down class, all its public methods are final. There is literally nothing to be gained by extending it. <br />
<br />
<a name="HowitWorksInjector"></a><h5>Injector</h5>The inversion of control mini-framework is fairly simple and has no advanced features. The injector injects values into fields marked by <code>@Inject</code> annotation. <br />
<br />
Values to be injected are created by object factories which are supplied to the injector in its constructor. Injected value depends on declared class or interface and nothing else. All fields of the same type obtain the value from the same factory.<br />
<br />
In addition, if you inject values into an object that implements <code>ObjectDecorator</code> interface, the framework will inject values also into the decorated object. Other than that, there are no conditions or any other advanced logic.<br />
<br />
<a name="HowitWorksWroManagerFactory"></a><h5>Wro Manager Factory</h5>Wro manager factory creates and configures wro manager. It must implement <code>WroManagerFactory</code> interface. <br />
<br />
Although you can create an entirely new implementation of <code>WroManagerFactory</code>, extending the default manager factory <code>BaseWroManagerFactory</code> or one of its sub classes makes more sense. Base factory does all that is necessary, supplies reasonable defaults wherever needed and is designed to be extensible.<br />
<br />
Each helper and factory object is created in its own <code>new...</code> method:<ul><li><code>newModelFactory</code> - factory to create the group model. Default model is configured in a <a href="#ConfigurationConfiguringGroupsConfigurationFileStructure">configuration file</a>.</li>
<li><code>newModelTransformers</code> - transformers are applied on created model. the default list contains only wildcard expander. It expands all resources containing <code>*</code> or <code>**</code> into list of concrete files.</li>
<li><code>newGroupExtractor</code> - extracts group name and requested resource type from the request uri. The default extractor takes file name as a group name and a suffix as a resource type.</li>
<li><code>newProcessorsFactory</code> - factory that creates all pre-processor and post-processors.</li>
<li><code>newHashBuilder</code> - computes hash used for caching and as an <a href="#OverviewOptimizationsEntityTag">etag code</a>. </li>
<li><code>newCacheStrategy</code> - cache is hidden behind this interface.</li>
<li><code>newUriLocatorFactory</code> - uri locator locates resources. It takes uri as an input and returns stream. There are three default uri locators: servlet contex locator, class path locator and url locator.</li>
</ul><br />
Base wro manager factory creates injector configured with all objects and factories created by <code>new...</code> methods. The injector is then used on those objects and on all objects created by <code>new...Factory</code> methods.<br />
<br />
The end result is that you can use the <code>@Inject</code> annotation to create dependencies between objects created in base wro manager factory and its child factories. <br />
<br />
<a name="Configuration"></a><h4>Configuration</h4>Once configured, wro4j requires very little maintenance. It is just there and transparently does whatever it needs to do. <br />
<br />
This chapter starts with small sub-chapters on Maven dependencies and on wro4j entry point servlet filter. <br />
<br />
It continues with groups configuration. We show how to create groups and add both JavaScript and CSS files into them. Once the groups are configured, wro4j is ready to be used. It does not compile less files yet, but all ordinary files are merged into groups and groups are served to the browser whenever needed. <br />
<br />
Finally, the last sub-chapter explains how to replace default pre/post-processors set with a custom one that contains also less compiler. <br />
<br />
<a name="ConfigurationMavenDependencies"></a><h5>Maven Dependencies</h5>Wro4j is split into multiple modules. We need two of them: wro4j-core and wro4j-extensions. Wro4j-core contains the minimum wro4j functionality and wro4j-extensions integrates less compiler. <br />
<br />
Add wro4j dependencies into pom.xml:<br />
<pre class="brush:xml"><properties>
<wro4j.version>1.4.6</wro4j.version>
</properties>
<dependency>
<groupId>ro.isdc.wro4j</groupId>
<artifactId>wro4j-core</artifactId>
<version>${wro4j.version}</version>
</dependency>
<dependency>
<groupId>ro.isdc.wro4j</groupId>
<artifactId>wro4j-extensions</artifactId>
<version>${wro4j.version}</version>
</dependency>
</pre><br />
Wro4j-extension manages its dependencies in non-traditional way. Its pom.xml states only the dependencies needed for all its features. Any additional one has to be included by your project.<br />
<br />
It does so because it integrates multiple compilers and libraries and any given project is going to use only small subset of them. There is no reason to download and link CoffeScript compiler with all its dependencies if you are not going to use it.<br />
<br />
Less compiler processor depends on Rhino, so we have to add it into pom.xml too:<br />
<pre class="brush:xml"><dependency>
<groupId>rhino</groupId>
<artifactId>js</artifactId>
<version>1.7R2</version>
</dependency>
</pre><br />
<a name="ConfigurationFilter"></a><h5>Filter</h5>The <code>WroFilter</code> servlet filter is the entry point into wro4j. The filter listens to browser requests and initiate all wro4j work when those requests come. <br />
<br />
It is configured the same way as any other filter in web.xml file. It listens to all requests for urls matching some pattern. For example, if you want it to place all wro4j groups on <code>/wro/</code> path, add following into web.xml file:<pre class="brush:xml"><filter>
<filter-name>WebResourceOptimizer</filter-name>
<filter-class>ro.isdc.wro.http.WroFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>WebResourceOptimizer</filter-name>
<url-pattern>/wro/*</url-pattern>
</filter-mapping>
</pre><br />
The filter should catch only requests for wro4j groups. It assumes that all requests are requests for some merged JavaScript or CSS group. If the assumption turns out to be wrong, wro4j logs an exception and passes the request further. The right file is eventually served, but the log file is polluted by unnecessary exceptions.<br />
<br />
<a name="ConfigurationConfiguringGroups"></a><h5>Configuring Groups</h5>Groups configuration is called a model. Model knows which files belong to each group and whether they should be minimized.<br />
<br />
Wro4j offers four options on how to configure it: <ul><li><a href="http://code.google.com/p/wro4j/wiki/WroFileFormat">wro.xml xml file</a>,</li>
<li><a href="http://code.google.com/p/wro4j/wiki/GroovyWroModel">wro.groovy groovy file</a>,</li>
<li><a href="http://code.google.com/p/wro4j/wiki/JsonWroModel">JSON configuration</a> - requires additional dependency,</li>
<li>use java to dynamically <a href="http://code.google.com/p/wro4j/wiki/WroModelRuntime">create groups</a>.</li>
</ul><br />
Xml, groovy and JSON configurations differ only in syntax. They offer the same possibilities and work the same way. This post show how to use xml version. <br />
<br />
Creating model dynamically from java is naturally more powerful and explained in the <a href="#DynamicModel">last chapter</a>.<br />
<br />
<a name="ConfigurationConfiguringGroupsConfigurationFileStructure"></a><h6>Configuration File Structure</h6>The xml configuration file must be named wro.xml and placed in WEB-INF directory. <br />
<br />
Its structure is very simple: the root element is <code>groups</code> tag. It contains a list of groups and each group is described by a <code>group</code> tag. Each group tag has a name and contains list of resources that belongs to it. <br />
<br />
Following xml contains one group called main. The group contains all css, less and JavaScript files from <code>/resources/</code> directory and its sub-directories:<br />
</div><pre class="brush:xml"><?xml version="1.0" encoding="UTF-8"?>
<groups xmlns="http://www.isdc.ro/wro"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.isdc.ro/wro wro.xsd">
<group name="main">
<css>/resources/**.css</css>
<css>/resources/**.less</css>
<js>/resources/**.js</js>
</group>
</groups>
</pre><div style="text-align: justify;"><br />
The above "everything in one group" configuration assumes that all style sheets are compatible with each other and the group is applicable to every page. Otherwise said, it is impossible to have different css theme for some pages. It also assumes that all JavaScripts are compatible with each and needed on every page. <br />
<br />
<a name="ConfigurationConfiguringGroupsConfigurationFileStructureMultiGroups"></a>If this is the case, then you have to split style sheets and JavaScripts into multiple groups:</div><pre class="brush:xml"><?xml version="1.0" encoding="UTF-8"?>
<groups xmlns="http://www.isdc.ro/wro"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.isdc.ro/wro wro.xsd">
<group name="theme1">
<css>/resources/common/**.css</css>
<css>/resources/common/**.less</css>
<js>/resources/common/**.js</js>
<css>/resources/theme1/**.css</css>
<css>/resources/theme1/**.less</css>
<js>/resources/theme1/**.js</js>
</group>
<group name="theme2">
<css>/resources/common/**.css</css>
<css>/resources/common/**.less</css>
<js>/resources/common/**.js</js>
<css>/resources/theme2/**.css</css>
<css>/resources/theme2/**.less</css>
<js>/resources/theme2/**.js</js>
</group>
</groups>
</pre><div style="text-align: justify;"><br />
<a name="ConfigurationConfiguringGroupsJsandCssTags"></a><h6>Js and Css Tags</h6>Resources are listed using <code>js</code> and <code>css</code> tags. Whatever is referenced by <code>js</code> tag is assumed to be JavaScript and whatever is referenced by <code>css</code> tag is assumed to be CSS file. <br />
<br />
It is important to keep this distinction clear. JavaScript and CSS files from the same group are served separately and use different pre and post processors.<br />
<br />
Both <code>js</code> and <code>css</code> tags can reference resources on classpath, file system, relative to the servlet context or by url. It is also possible to extend them and add new resource locations (database, ...), but such thing is out of scope of this post.</div><pre class="brush:xml"><css>classpath:org/somewhere/style.css</css>
<css>file:src/main/webapp/resources/hangman/style.css</css>
<css>http://localhost:8080/wicket-wro/resources/hangman/style.css</css>
</pre><div style="text-align: justify;"><br />
They both support <code>*</code> wildcard and <code>**</code> deep wildcard. Wildcard matches only files in specified directory. It does not go into sub-directories. Deep wildcard matches also files in all sub-directories: </div><pre class="brush:xml"><group name="deep_wildcard_goes_into_subdirectories">
<css>/resources/**.css</css>
</group>
<group name="wildcard_has_to_list_all_subdirectories">
<css>/resources/*.css</css>
<css>/resources/hangman/*.css</css>
<css>/resources/home/*.css</css>
</group>
</pre><div style="text-align: justify;"><br />
All resources in the group are minimized by default. If you wish to turn off minimization for some resource, add <code>minimize="false"</code> attribute to <code>js</code> or <code>css</code> tag. Of course, the <code>minimize</code> attribute works only if the minimizing processor is configured as a pre-processor (more on that <a href="#ConfigurationProcessors">later</a>).</div><pre class="brush:xml"><css minimize="false">/resources/**.css</css>
</pre><div style="text-align: justify;"><br />
<a name="ConfigurationConfiguringGroupsJsandCssTagsBug"></a>Note: Both <code>js</code> and <code>css</code> tags in the current version 1.4.6 have <a href="https://github.com/alexo/wro4j/pull/44">a bug</a>. Neither of them works correctly if it references the project root. For example, <code><css>/**.css</css></code> does not work. The workaround is to place all resources in some sub-directory and reference that one: <code><css>/resources/**.css</css></code>. The bug was already fixed and the next released version will not have this limitation.<br />
<br />
<a name="ConfigurationConfiguringGroupsGroupReferences"></a><h6>Group References</h6>Any group can reference resources from other groups using the <code>group-ref</code> tag. <br />
<br />
It is used to avoid repetition and place common resources into one group. For example, previous <a href="#ConfigurationConfiguringGroupsConfigurationFileStructureMultiGroups">multi-group configuration</a> can be rewritten into following form: </div><pre class="brush:xml"><?xml version="1.0" encoding="UTF-8"?>
<groups xmlns="http://www.isdc.ro/wro"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.isdc.ro/wro wro.xsd">
<group name="common-resources">
<css>/resources/common/**.css</css>
<css>/resources/common/**.less</css>
<js>/resources/common/**.js</js>
</group>
<group name="theme1">
<group-ref>common-resources</group-ref>
<css>/resources/theme1/**.css</css>
<css>/resources/theme1/**.less</css>
<js>/resources/theme1/**.js</js>
</group>
<group name="theme2">
<group-ref>common-resources</group-ref>
<css>/resources/theme2/**.css</css>
<css>/resources/theme2/**.less</css>
<js>/resources/theme2/**.js</js>
</group>
</groups>
</pre><div style="text-align: justify;"><br />
<a name="ConfigurationUsingGroups"></a><h5>Using Groups</h5>Configured groups are used the same way as any other JavaScript or css resource. <br />
<br />
Assuming that the wro filter catches every request for <code>/wro/*</code> and that you have a group named <code>main</code>, you can include all its css resources using: <br />
</div><pre class="brush:html"><link rel="stylesheet" type="text/css" href="wro/main.css">
</pre><div style="text-align: justify;"><br />
Similarly, you can include all its js resources the same way as you would an ordinary JavaScript: <br />
</div><pre class="brush:html"><script type="text/javascript" src="wro/main.js"></script></pre><div style="text-align: justify;"><br />
Note: unless configured <a href="#DynamicModelGroupExtractor">otherwise</a>, wro4j takes requested file name as a group name and a suffix as a resource type. It ignores the path and recognizes only two possible suffixes: .css and .js. Therefore, following snippet loads the same group as previous two: </div><pre class="brush:html"><script type="text/javascript" src="wro/something/main.js"></script>
<link rel="stylesheet" type="text/css" href="wro/something/main.css">
</pre><div style="text-align: justify;"><br />
<a name="ConfigurationProcessors"></a><h5>Processors</h5>Wro4j applies pre-processors to each file before merging and post-processors to the merged group. As we have not explicitly configured them yet, our project is using the default set.<br />
<br />
The list of processors is created in wro manager factory. The default factory supplies reasonable list of default processors, but it is unable to create less compiler. Therefore, we have to: <ul><li>replace the default factory with something more flexible,</li>
<li>configure new factory to use all necessary pre/post-processors,</li>
<li>configure new factory to use less compiler.</li>
</ul><br />
First section shows how to replace the default factory and second chapter explains how to configure it. Remaining sections go through all used processors and explain what they do and why we had to use them. <br />
<br />
<a name="ConfigurationProcessorsReplaceManagerFactory"></a><h6>Replace Manager Factory</h6>First we have to choose the right manager factory. Manager factory is a class that implements <code>WroManagerFactory</code> interface and wro4j has a lot of them. Most of manager factory implementations serve unit tests, Maven integration, command line integration or other "helper" cases.<br />
<br />
Four manager factories are designed to be used in web application:<ul><li><code>DefaultWroManagerFactory</code> - used if nothing else is configured.</li>
<li><code>CustomWroManagerFactory</code> - meant to be extended. Extend it if no standard factory produces what you need. </li>
<li><code>ConfigurableWroManagerFactory</code> - able to create all processors from basic wro4j package. Partially configurable in wro.properties file.</li>
<li><code>ExtensionsConfigurableWroManagerFactory</code> - able to create all processors from wro4j-extensions package. Partially configurable in wro.properties file.</li>
</ul><br />
We will use <code>ExtensionsConfigurableWroManagerFactory</code> because it supports all processors we need and is easy to configure. <br />
<br />
Create wro.properties file and place it to the WEB-INF directory. Use <code>managerFactoryClassName</code> property to configure wro manager factory class name:<br />
<pre class="brush:ps">managerFactoryClassName=ro.isdc.wro.extensions.manager.ExtensionsConfigurableWroManagerFactory
</pre><br />
<a name="ConfigurationProcessorsConfigureManagerFactory"></a><h6>Configure Manager Factory</h6>Extension configurable manager factory has three configurable properties:<ul><li><code>uriLocators</code> - uri locators locates resources. They take uri as an input and returns stream. This property has reasonable default value, so we will leave it as it is.</li>
<li><code>preProcessors</code> - list of pre-processors. It is empty by default.</li>
<li><code>postProcessors</code> - list of post-processors. It is empty by default.</li>
</ul><br />
Neither merged .css nor merged .js files work correctly with empty pre-processors list. Simplest working configuration requires three pre-processors: <code>cssUrlRewriting</code>, <code>cssImport</code> and <code>semicolonAppender</code>.<br />
<br />
Post-processors are not necessary, but some of them are very useful. We decided to use three post-processors: <code>lessCss</code>, <code>jsMin</code> and <code>cssMin</code>.<br />
<br />
Final wro.properties file:<br />
<pre class="brush:ps">managerFactoryClassName=ro.isdc.wro.extensions.manager.ExtensionsConfigurableWroManagerFactory
preProcessors=cssUrlRewriting,cssImport,semicolonAppender
postProcessors=lessCss,jsMin,cssMin
</pre><br />
Processors are applied in the configured order. Each processor knows whether it should be applied to JavaScript, css or both files. You do not have to worry about wrong type of processor being applied to wrong type of file.<br />
<br />
<a name="ConfigurationProcessorsRelativeReferencesinCSS"></a><h6>Relative References in CSS</h6>Loading css file from the group instead of its original location breaks relative references. Unless import statements and image locations are rewritten, <code>@import</code> statement or referenced images do not work anymore.<br />
<br />
Resources references rewriting is done in <code>cssUrlRewriting</code> and <code>cssImport</code> pre-processors. Their order does matter, the css url rewriting processor must go first.<br />
<br />
The <code>cssUrlRewriting</code> rewrites relative references to images so they work from the new location. The <code>cssImport</code> does two things:<ul><li>adds imported .css files into the group,</li>
<li>removes the import statement.</li>
</ul><br />
Use both <code>cssUrlRewriting</code> and <code>cssImport</code> as pre-processors:<br />
<pre class="brush:ps">preProcessors=cssUrlRewriting,cssImport
</pre><br />
<a name="ConfigurationProcessorsMissing;in.jsFiles"></a><h6>Missing ; in .js Files</h6>JavaScript file may miss last semicolon and still be valid JavaScript file. However, once you merge those files together, the missing semicolon may cause a lot of problems. <br />
<br />
Use <code>semicolonAppender</code> pre-processor to add missing last <code>;</code> into JavaScript files. <br />
<br />
Add <code>semicolonAppender</code> to pre-processors list:<br />
<pre class="brush:ps">preProcessors=cssUrlRewriting,cssImport,semicolonAppender
</pre><br />
<a name="ConfigurationProcessorsMinimization"></a><h6>Minimization</h6>Strictly speaking, this is not necessary. However, since we already added wro4j into the project and configured it, we can take advantage of its minimization abilities as well.<br />
<br />
Use <code>cssMin</code> to minimize css files and <code>jsMin</code> to minimize JavaScript files. Both are able to work as both pre-processors and post-processors.<br />
<pre class="brush:ps">postProcessors=jsMin,cssMin
</pre><br />
Note: jsMin is not the only JavaScript minimizer available in wro4j. Wro4j-extensions contains also Google closure compiler, YUI compression utilities, Dojo Shrinksafe utility and other popular minimizers. All minimizers are listed in the <a href="http://code.google.com/p/wro4j/wiki/AvailableProcessors">processors list</a>.<br />
<br />
<a name="ConfigurationProcessorsLessCompiler"></a><h6>Less Compiler</h6>Less compiler is implemented in <code>LessCssProcessor</code>. It can act as both pre-processor or post-processor. If you use it as a pre-processor the <code>@import</code> statement does not work, so it is better to ignore that option and use it as a post-processor only. <br />
<br />
Do not worry about JavaScript files in the same group, LessCssProcessor does not modify them. <br />
<br />
Add less compiler to the list of post processors. It make more sense to compile first and minimize later:<br />
<pre class="brush:ps">postProcessors=lessCss,jsMin,cssMin
</pre><br />
<a name="DynamicModel"></a><h4>Dynamic Model</h4>Having files grouped together lowers number of requests, but also adds some additional hustle. The more fine grained groups are, the more likely they break when you add, move, rename or remove files. <br />
<br />
If the model is predictable, it can be build dynamically in Java code. If it is done right, wro4j acts transparently to the rest of the application and requires minimal maintennance. <br />
<br />
This chapter shows how to create very simple dynamic model: each .css and .less file have its own group and is processed separately. As an additional bonus, it fixes the bug mentioned <a href="#ConfigurationConfiguringGroupsJsandCssTagsBug">above</a>.<br />
<br />
<a name="DynamicModelSampleProject"></a><h5>Sample Project</h5>We took Apache Wicket Hangman example project from the Apache repository and modified it to use wro4j with dynamic model.<br />
<br />
Use the <code>StartExamples</code> class to run the project. It will configure and start jetty server with hangman application running on it. The application is then available at <code>http://localhost:8080/wicket-wro/hangman/</code> url.<br />
<br />
The <a href="https://github.com/SomMeri/less-wro-wicket">sample project</a> is available on Github.<br />
<br />
<a name="DynamicModelFilter"></a><h5>Filter</h5>First, we have to configure WroFilter to catch all requests for .css or .less files. We will not assume that they are all located in some special location.<br />
<br />
Add following into web.xml file:<br />
<pre class="brush:xml"><filter>
<filter-name>WebResourceOptimizer</filter-name>
<filter-class>ro.isdc.wro.http.WroFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>WebResourceOptimizer</filter-name>
<url-pattern>*.less</url-pattern>
</filter-mapping>
<filter-mapping>
<filter-name>WebResourceOptimizer</filter-name>
<url-pattern>*.css</url-pattern>
</filter-mapping>
</pre><br />
<a name="DynamicModelTheModel"></a><h5>The Model</h5>Wro4j uses two steps algorithm to create the model: <ul><li>model factory creates initial model,</li>
<li>initial model is transformed into final model using model transformers.</li>
</ul><br />
The default model factory creates initial model that corresponds exactly to the content of wro.xml file. Wro4j then applies its only default model transformer to it. The transformer is called wildcard model transformer and expands all resources containing * or ** into list of concrete file resources.<br />
<br />
We have to create a model that contains all .css and .less files, so it makes sense to take advantage of the wilcard model transformer. Our initial model has only one group which uses deep wilcard to reference all those files. It corresponds to following xml file: <pre class="brush:xml"><group name="fake">
<css>/**.css</css>
<css>/**.less</css>
</group>
</pre><br />
Then we use wildcard model transformer to replace wildcards with list of concrete files. Once we have all files in one group, we will use custom model transformer to move each file into separate group.<br />
<br />
<a name="DynamicModelTheModelInitialModel"></a><h6>Initial Model</h6>The model must implement <code>WroModel</code> interface and is created by an instance of <code>WroModelFactory</code>. The model contains a list of groups <code>Group</code> and each group contains list of resources <code>Resource</code>:<br />
<pre class="brush:java">public class ConfigurationFreeModelFactory implements WroModelFactory {
public WroModel create() {
WroModel result = new WroModel();
result.addGroup(createAllResourcesGroup());
return result;
}
private Group createAllResourcesGroup() {
Resource allLess = new Resource();
allLess.setType(ResourceType.CSS);
allLess.setUri("/**.less");
Resource allCss = new Resource();
allCss.setType(ResourceType.CSS);
allCss.setUri("/**.css");
Group group = new Group("fake");
group.addResource(allCss);
group.addResource(allLess);
return group;
}
public void destroy() {
}
}
</pre><br />
<a name="DynamicModelTheModelModelTransformers"></a><h6>Model Transformers</h6>We use two model transformers:<ul><li>wildcard model transformer to expand wildcards,</li>
<li>custom resources to groups model transformer to create a separate group for each file.</li>
</ul><br />
Wildcard model transformer expands wildcards into list of resources. The default wildcard model transformer has a <a href="#ConfigurationConfiguringGroupsJsandCssTagsBug">bug</a>, so we will replace it with a fixed one. <br />
<br />
Find the <a href="https://github.com/SomMeri/less-wro-wicket/blob/master/src/main/java/org/meri/wro4j/nogroups/fix/FixedWildcardExpanderModelTransformer.java#L34">fixed model transformer</a> in our demo project on Github. Since the bug is already fixed in wro4j trunk, we will not show the code here. The solution is only temporary and fixed wildcard model transformer will not be needed after the next wro4j release. <br />
<br />
Second model transformer creates a separate group for each resource. Each group must have unique name, so we used resource uri as the group name. For example, a style sheet named <code>main.css</code> and located in <code>resources</code> directory is placed into group named <code>/resources/main.css</code>:<br />
</div><pre class="brush:java">public class ResourcesToGroupsModelTransformer implements Transformer<WroModel> {
public WroModel transform(WroModel input) throws Exception {
WroModel result = new WroModel();
for (Group group : input.getGroups()) {
for (Resource resource : group.getResources()) {
Group resourceGroup = toGroup(resource);
result.addGroup(resourceGroup);
}
}
return result;
}
private Group toGroup(Resource resource) {
Group resourceGroup = new Group(resource.getUri());
resourceGroup.addResource(resource);
return resourceGroup;
}
}
</pre><div style="text-align: justify;"><br />
<a name="DynamicModelGroupExtractor"></a><h5>Group Extractor</h5>Group extractor extracts group name and resource type out of incoming request. It also returns whether the group should be minimized and whether it is possible to compose group name and resource type back to the original uri.<br />
<br />
Default group extractor takes requested file name as the group name and a suffix as the resource type. It recognizes only two possible suffixes: .css and .js. Our dynamic model uses whole uri as the group name and contains also .less files. As a result, the default wro configuration is unable to find any of our generated groups.<br />
<br />
For example, the request for <code>context path/resource/main.css</code> would be resolved as a request for all style sheets inside the <code>main</code> group. As our group is named <code>/resource/main.css</code>, it would not be found. Additionally, we need it to be recognize .less files as a style sheets, but the default implementation discards them as unknown resource types.<br />
<br />
A group extractor must implement <code>GroupExtractor</code> interface. Its <a href="https://github.com/SomMeri/less-wro-wicket/blob/master/src/main/java/org/meri/wro4j/nogroups/ConfigurationFreeGroupExtractor.java#L25">full implementation</a> is available on Github. Most of it is trivial, so following example contains only the most important part:<br />
</div><pre class="brush:java">public class ConfigurationFreeGroupExtractor implements GroupExtractor {
private final DefaultGroupExtractor defaultExtractor =
new DefaultGroupExtractor();
/**
* Everything that follows context path is considered a group name.
*/
public String getGroupName(HttpServletRequest request) {
String contextPath = request.getContextPath();
String uri = getUri(request);
if (uri.startsWith(contextPath))
uri = uri.substring(contextPath.length());
return uri;
}
/**
* If the default extractor is unable to find the resource type,
* check whether it is a .less file. Less files are considered
* style sheets.
*/
public ResourceType getResourceType(HttpServletRequest request) {
ResourceType resourceType =
defaultExtractor.getResourceType(request);
//if the default extractor could not find the type
//check whether it is .less file
if (resourceType==null && isLessFile(request)) {
resourceType = ResourceType.CSS;
}
return resourceType;
}
private boolean isLessFile(HttpServletRequest request) {
return request.getRequestURI().toUpperCase().endsWith(".LESS");
}
...
}
</pre><div style="text-align: justify;"><br />
<a name="DynamicModelWroManagerFactory"></a><h5>Wro Manager Factory</h5>We have custom model factory, custom group extractor and custom model transformer. The last thing we need is to create and configure custom wro manager factory that produces our new objects. <br />
<br />
All wro manager factories are designed in very similar way and it does not matter too much which one we extend. As we have ExtensionsConfigurableWroManagerFactory already configured with needed processors, we will extend this one and override three its methods:<br />
<ul><li><code>newModelFactory</code> - creates model factory,</li>
<li><code>newModelTransformers</code> - creates model transformers,</li>
<li><code>newGroupExtractor</code> - creates group extractor.</li>
</ul><br />
New factory is located in <code>org.meri.wro4j.nogroups</code> package:</div><pre class="brush:java">public class ConfigurationFreeManagerFactory extends ExtensionsConfigurableWroManagerFactory {
@Override
protected GroupExtractor newGroupExtractor() {
return new ConfigurationFreeGroupExtractor();
}
@Override
protected WroModelFactory newModelFactory() {
return new ConfigurationFreeModelFactory();
}
@Override
protected List<Transformer<WroModel>> newModelTransformers() {
//The super class store the list of model transformers in a private
//property. We need both modify that property and return the list of
//correct transformers.
List<Transformer<WroModel>> modelTransformers =
super.newModelTransformers();
//replace default WildcardExpanderModelTransformer with
//FixedWildcardExpanderModelTransformer
modelTransformers.clear();
addModelTransformer(new FixedWildcardExpanderModelTransformer());
addModelTransformer(new ResourcesToGroupsModelTransformer());
return modelTransformers;
}
}
</pre><div style="text-align: justify;"><br />
Wro.properties configuration:<pre class="brush:text">managerFactoryClassName=org.meri.wro4j.nogroups.ConfigurationFreeManagerFactory
preProcessors=cssUrlRewriting,cssImport,semicolonAppender
postProcessors=lessCss,jsMin,cssMin</pre><br />
<a name="End"></a><h4>End</h4>Although wro4j configuration requires some work, easy access to optimizations and technologies like less or CoffeScript may be worth it.<br />
<br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com15tag:blogger.com,1999:blog-3355333693246614846.post-12003123165005532742012-07-01T08:00:00.002-07:002013-03-31T01:26:29.675-07:00Inheritance and Variables in CSS with Less<style type="text/css">div.blogger-clickTrap {display: none;}</style><div style="text-align: justify;">I always missed object oriented inheritance and variables in CSS. Style sheets could be much more readable and maintainable, if they would have ability to say: <br />
<ul><li>I want this selector color to be just like that one.</li>
<li>I want this class to be just like that one, with one simple change.</li>
</ul><br />
As multiple projects run into <a href="http://stackoverflow.com/questions/2253110/how-to-manage-css-explosion">CSS explosion</a> and <a href="http://stackoverflow.com/questions/5419966/how-can-i-stay-organized-when-writing-css">maintenance problems</a> problems, it was only question of time until someone comes with another solution. The <a href="http://sass-lang.com/">original solution</a> came from the Ruby world and is quite simple and elegant:<ul><li>extend CSS with needed features,</li>
<li>compile extended CSS into regular CSS,</li>
<li>serve the result to the browser.</li>
</ul><br />
The browser does not have to deal with a new syntax. It was given standard CSS file and will process it as usually. On the other hand, the programmer had variables, inheritance and few other features available.<a name='more'></a><br />
<br />
This post introduces multiple extended CSS languages, picks up one and shows how to use it. The next post shows how to use wro4j library to integrate the compiler into java web project. <br />
<br />
<h4>Extended CSS Languages</h4>At least three different extended CSS languages are available: <ul><li><a href="http://sass-lang.com/">SASS</a>,</li>
<li><a href="http://learnboost.github.com/stylus/">Stylus</a>,</li>
<li><a href="http://lesscss.org/">LESS</a>.</li>
</ul><br />
We decided to use LESS, because its integration into Java project is easiest. Its compiler was written in JavaScript and is able to run either directly in the browser or on the server using <a href="http://www.mozilla.org/rhino/">Rhino</a>.<br />
<br />
Stylus was also written in JavaScript, but was meant for node.js environment. Slightly modified version <a href="http://learnboost.github.com/stylus/try.html">can run</a> either from Java or directly in the browser. The browser compliant solution is part of <a href="https://github.com/halfbaked/grails-stylus-resources">grails plugin</a>. If you want to use it, you can dug it out of there.<br />
<br />
SASS was the first extended CSS language out there and its compiler was written in ruby. It had also an alternative JavaScript compiler <a href="https://github.com/visionmedia/sass.js">sass.js</a>, but the project is not maintained anymore. Last commit was done 2 years ago and its owner started the Stylus project.<br />
<br />
If you want to read more about these languages, an excellent <a href="net.tutsplus.com/tutorials/html-css-techniques/sass-vs-less-vs-stylus-a-preprocessor-shootout">side-by-side comparison</a> is available on net.tuts+ site. <br />
<br />
<h4>Less Introduction</h4>LESS was originally developed by <a href="http://cloudhead.io/">Alexis Sellier alias cloudhead</a>. It is still actively maintained and developed. The project has also multiple committers and numerous contributors. <br />
<br />
Its <a href="https://github.com/cloudhead/less.js">source code</a> is hosted on Github and distributed under Apache License 2.0 license.<br />
<br />
This chapter shows how to create a simple project that compiles .less files during the page load on the browser. That simple project is then used to show the most important less features.<br />
<br />
<h5>Sample Project</h5>Less can run entirely in the browser, so you can experiment with it on a simple HTML page. Running it this way is practical for small HTML and JavaScript projects or if you have not decided whether you want to use it or not. <br />
<br />
Of course, running the compiler in the browser slows down the page load and does not work if the JavaScript is disabled. If one of these is a concern, then you will have to move to compiler to the <a href="#ServerSideIntegration">server side</a>.<br />
<br />
<h6>Overview</h6>Using less on the browser side is quite simple:<ul><li>Load all .less style sheets using <code>link</code> tag.</li>
<li>Load the compiler using <code>script</code> tag.</li>
</ul><br />
All .less files have to be loaded before the compiler script is loaded:<br />
</div><pre class="brush:html"><head>
<link rel="stylesheet/less" type="text/css" href="main-theme.less">
...
<link rel="stylesheet/less" type="text/css" href="other-theme.less">
<script src="less.min.js" type="text/javascript"></script>
</head>
</pre><div style="text-align: justify;"><br />
<h6>Sample .less File</h6>Start with the sample .less file from the less <a href="http://lesscss.org/">main page</a>. It has enough of less features in it, so we can use it to test whether the less compiler works.<br />
<br />
Create new directory and place theme.less file in it:<br />
<pre class="brush:css">@base: #38b9ab;
.box-shadow(@style, @c) when (iscolor(@c)) {
box-shadow: @style @c;
-webkit-box-shadow: @style @c;
-moz-box-shadow: @style @c;
}
.box-shadow(@style, @alpha: 50%) when (isnumber(@alpha)) {
.box-shadow(@style, rgba(0, 0, 0, @alpha));
}
.box {
color: saturate(@base, 5%);
border-color: lighten(@base, 30%);
div { .box-shadow(0 0 5px, 30%) }
}
</pre><br />
<h6>Install Less Compiler</h6><a href="http://lesscss.org/">Download less.js</a> and place it into the same directory. Rename it to less.min.js.<br />
<br />
<h6>Sample HTML File</h6>Create sample html file and place it to the same directory. Our sample html file loads theme.less file:<br />
</div><pre class="brush:html"><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link rel="stylesheet/less" type="text/css" href="theme.less">
<script src="less.min.js" type="text/javascript"></script>
<title>Less Demo</title>
</head>
<body>
<div class="box">
Box: both text and border colors are functions of the @base variable. <br> The border is 30% lighter and the text is 5% saturated.
<div>
A div inside a box has a grey shadow.
</div>
</div>
<p>No CSS available for this element.</p>
</body>
</html>
</pre><div style="text-align: justify;"><br />
<h6>Test It</h6>Open HTML file in the browser. Less compiler compiles .less style sheet during the page load and compiled css is applied to the HTML. <br />
<br />
Everything inside the 'box' div is blue and the inner div has grey shadow.<br />
<br />
<h5>Less Basics</h5>This sub-chapter shows the most important less features. We are not going to go into details, less <a href="http://lesscss.org/#docs">documentation</a> does a good job at that. We describe only what is needed to understand .less file used in previous chapter. <br />
<br />
Almost all sections show one less feature and contains less style sheet example. Example files are compatible with simple project created in previous chapter. You can replace theme.less file content with any one and see how the page changes.<br />
<br />
All examples used in this chapter are available in a Github repository named <a href="https://github.com/SomMeri/less.js-demo">less.js-demo</a>.<br />
<br />
<h6>Error Reporting</h6>If the .less file is invalid, Less compiler shows incorrect line along with its line number on top of the screen. This is makes debugging style sheets much easier.<br />
<br />
<h6>CSS Compatibility</h6>Less extends CSS and makes style sheets more dynamic. Any valid .css file is also a valid .less file. <br />
<br />
Less compiler supports most <a href="http://www.ejeliot.com/blog/63">style sheet hacks</a>, but not necessarily all of them. For example, IE 6 only hack from the style sheet hacks article does not work:<br />
<pre class="brush:css">_bac\kground: #f60;
</pre><br />
The good new is that whenever someone demanded a new hack, the project maintainer responded fast and added missing hacks into the less. Therefore, if you need the above construction for some reason, it is quite possible that you are going to get it.<br />
<br />
Note: we did not reported the missing IE 6 only hack as a bug, because we do not know whether someone actually needs it.<br />
<br />
<h6>Nesting</h6>Nesting means, that a rule can be placed inside another rule. The main advantage of this construct is a scope. Nested rule can use any variable or <a href="#Mixins">mixin</a> defined inside its parent rules. Both variables and mixins are explained in following sections.<br />
<br />
If we forget about scoping, rule nesting is just another way how to express descendant selectors. A simple nested rule: </div><pre class="brush:css">.box {
/* box properties */
display:inline-block;
padding: 4px;
div {
/* div inside box properties */
padding-left: 4px;
padding-right: 4px;
}
}
</pre><div style="text-align: justify;"><br />
is equivalent to following .css: </div><pre class="brush:css">.box {
/* box properties */
display:inline-block;
padding: 4px;
}
.box div {
/* div inside box properties */
padding-left: 4px;
padding-right: 4px;
}
</pre><div style="text-align: justify;"><br />
<h6>Variables</h6>Variables work the same way as constants in any other language. They serve as a symbolic name for a value that can not change. They are prefixed with <code>@</code> and usable only within a block where they have been declared. <br />
<br />
The next snipped shows how to use variables to define consistent padding. The <code>.box</code> class defines a local <code>@defPad</code> variable. It contains default padding size and is usable only inside the <code>.box</code> and its nested <code>.box div</code> rule. New example contains also a global variable <code>@base</code> which could be used anywhere.<br />
<br />
The example is inside theme-variables.less file:<br />
</div><pre class="brush:css">/* unused global variable */
@base: #38abf9;
.box {
/* local variable */
@defPad: 4px;
display:inline-block;
padding: @defPad;
div {
/* div inside box properties */
padding-left: @defPad;
padding-right: @defPad;
}
}
/* @defPad is NOT available here */
</pre><div style="text-align: justify;"><br />
The compiled style sheet:<br />
</div><pre class="brush:css">.box {
display: inline-block;
padding: 4px;
}
.box div {
padding-left: 4px;
padding-right: 4px;
}
</pre><div style="text-align: justify;"><br />
<h6>Functions and Operators</h6>Operators and functions add computational power to style sheets. You can use them to define a color to be slightly lighter than another one or padding twice as big as another one.<br />
<br />
Each operator works on both numbers and colors. Functions usually work with either numbers or colors. <ul><li>operators: +, - *, /,</li>
<li>math functions: round, ceil, floor, percentage,</li>
<li>color functions: darken, desaturate, mix, ... .</li>
</ul><br />
The complete list of color functions is too long and available in less documentation. <br />
<br />
The following code uses color functions to define similar colors for border, text and inner elements. The code is located in theme-functions-operators.less file:<br />
</div><pre class="brush:css">@base: #38abf9;
.box {
border: 2px solid lighten(@base, 30%);
color: saturate(@base, 5%);
div {
color: fade(@base, 30%)
}
}
</pre><div style="text-align: justify;"><br />
The compiled style sheet:<br />
</div><pre class="brush:css">.box {
border: 2px solid #CDEAFD;
color: #33ACFE;
}
.box div {
color: rgba(56, 171, 249, 0.3);
}
</pre><div style="text-align: justify;"><br />
<a name="Mixins"></a><h6>Mixins</h6>Mixins are css rules that can be reused inside other rules. Use them to avoid repetition when you need the same set of properties inside multiple rules. <br />
<br />
For example, each browser uses different property to paint shadow around an element. We placed all three browser dependent properties into the <code>box-shadow</code> rule. The rule is used to add shadow to both <code>.box</code> and <code>.box div</code>.<br />
<br />
The example is located inside the theme-simple-mixin.less file:<br />
</div><pre class="brush:css">.box-shadow {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
.box {
.box-shadow;
div {
.box-shadow;
}
}
</pre><div style="text-align: justify;"><br />
The compiled style sheet:<br />
</div><pre class="brush:css">.box-shadow {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
.box {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
.box div {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
</pre><div style="text-align: justify;"><br />
As you can see, the mixin was copied inside the other rules. Other than that, it was treated like any other rule in the style sheet and copied into the resulting style sheet. If you do not want to have it there and want to treat it only as a helper rule, place () after the mixin declaration.<br />
<br />
The example is located inside the theme-clean-mixin.less file:<br />
</div><pre class="brush:css">.box-shadow() {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
.box {
.box-shadow;
div {
.box-shadow;
}
}
</pre><div style="text-align: justify;"><br />
The compiled style sheet does not contain <code></code> class:<br />
</div><pre class="brush:css">.box {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
.box div {
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
</pre><div style="text-align: justify;"><br />
<h6>Parametrized Mixins</h6>A mixin can be parametrized. Parameters do not have types and act as local variables inside the mixin. Mixin can have any number of parameters and each can have a default value. <br />
<br />
Note: the theme-clean-mixin from the previous section is an example of a parametrized mixin with empty list of variables.<br />
<br />
The <code>.box-shadow</code> mixin in the next example uses <code>@style</code> and <code>@color</code> parameters to customize shadows around various elements. The default <code>@color</code> is black. The final theme-parametrized-mixin.less file adds green shadow around the <code>.box</code>: and red shadow around the <code>.box div</code>:<br />
</div><pre class="brush:css">.box-shadow(@style, @color: rgb(0, 0, 0)) {
box-shadow: @style @color;
-webkit-box-shadow: @style @color;
-moz-box-shadow: @style @color;
}
.box {
.box-shadow(0 0 7px, rgba(0, 90, 0, 0.9));
div {
.box-shadow(0 0 7px, rgba(90, 0, 0, 0.9));
}
}
</pre><div style="text-align: justify;"><br />
The compiled style sheet:<br />
</div><pre class="brush:css">.box {
box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
-webkit-box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
-moz-box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
}
.box div {
box-shadow: 0 0 7px rgba(90, 0, 0, 0.9);
-webkit-box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
-moz-box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
}
</pre><div style="text-align: justify;"><br />
<h6>Guarded Mixins</h6>Guards are conditions attached to mixins. A guarded mixin is used only if its condition is satisfied. This allows you to have multiple mixins with the same name and number of parameters. It also simulates if statement.<br />
<br />
Guards conditions are quite powerful and ofter a lot of possibilities. They can use all standard math and color functions and operators. They can use also a set of boolean operators and functions (less then, equals, isColor, isNumber, ...). <br />
<br />
Guard is a keyword <code>when</code> followed by a condition. Most of them simply check mixin parameters types. <br />
<br />
The next example defines two mixins with the same name and same number of parameters. One is used if the second parameter is color and another is used if the second parameter is number: <br />
</div><pre class="brush:css">/* define shadow with supplied color */
.box-shadow(@style, @c) when (iscolor(@c)) {
box-shadow: @style @c;
-webkit-box-shadow: @style @c;
-moz-box-shadow: @style @c;
}
/* define black shadow with supplied transparency */
.box-shadow(@style, @alpha: 50%) when (isnumber(@alpha)) {
.box-shadow(@style, rgba(0, 0, 0, @alpha));
}
.box {
.box-shadow(0 0 7px, rgba(0, 90, 0, 0.9));
div {
.box-shadow(0 0 7px, 30%);
}
}
</pre><div style="text-align: justify;"><br />
The compiled style sheet:<br />
</div><pre class="brush:css">.box {
box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
-webkit-box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
-moz-box-shadow: 0 0 7px rgba(0, 90, 0, 0.9);
}
.box div {
box-shadow: 0 0 7px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 7px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 7px rgba(0, 0, 0, 0.3);
}
</pre><div style="text-align: justify;"><br />
<h6>All in One</h6>The following code has all the above features in it. In fact, it is slightly changed showcase taken from the less <a href="http://lesscss.org/">homepage</a>. <br />
<pre class="brush:css">@base: #38abf9;
.box-shadow(@style, @c) when (iscolor(@c)) {
box-shadow: @style @c;
-webkit-box-shadow: @style @c;
-moz-box-shadow: @style @c;
}
.box-shadow(@style, @alpha: 50%) when (isnumber(@alpha)) {
.box-shadow(@style, rgba(0, 0, 0, @alpha));
}
.box {
@defPad: 4px;
display:inline-block;
border: 2px solid lighten(@base, 30%);
color: saturate(@base, 5%);
padding: @defPad;
div {
padding-left: @defPad;
padding-right: @defPad;
.box-shadow(0 0 5px, 30%);
}
}
</pre><br />
It is compiled into following .css file:<br />
<pre class="brush:css">.box {
display: inline-block;
border: 2px solid #cdeafd;
color: #33acfe;
padding: 4px;
}
.box div {
padding-left: 4px;
padding-right: 4px;
box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-webkit-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
-moz-box-shadow: 0 0 5px rgba(0, 0, 0, 0.3);
}
</pre><br />
<a name="ServerSideIntegration"></a><h4>Server Side Integration</h4>Running less compiler in the browser is fine as long as the page load speed does not matter. Once the speed becomes important, it is better to move the compiler on the server.<br />
<br />
First sub-chapter lists links to resources useful if you want to run less.js from the command line. The following one contains quick overview of Java libraries that help to bring less.js on the server side.<br />
<br />
<h5>Command Line</h5>Less.js runs on node.js. If you have node.js installed on *nix system, you can use <a href="https://github.com/cloudhead/less.js/tree/master/bin">lesscss script</a> to run it from command line.<br />
<br />
Windows users can avoid node.js installation using the <a href="https://github.com/duncansmart/less.js-windows">less.js-windows</a> project. It is a handy command line wrapper over less.js and uses built-in Windows JavaScript runner. <br />
<br />
<h5>In Java</h5>As of now, less has only JavaScript compiler. There are two initiatives <a href="https://github.com/SomMeri/less4j">to create</a> a <a href="https://github.com/alexo/less4java">pure java compiler</a>, but they are in very early stage. The standard way is to use <a href="http://www.mozilla.org/rhino/">Rhino</a> as a JavaScript engine and run less.js in it.<br />
<br />
There are multiple libraries that already do that. If you aim for simplicity, you may use <a href="http://www.asual.com/lesscss/">lescss</a>. Another solution is in <a href="http://www.richardnichols.net/2010/06/less-css-in-wicket-using-mozilla-rhino/">visural-common and visural-wicket</a> libraries and Rhino based compiler-only project <a href="https://github.com/marceloverdijk/lesscss-java">exists too</a>.<br />
<br />
Finally, any Java project can use <a href="http://alexo.github.com/wro4j/">wro4j</a>. It is quite a large library that integrates not only less.js, but also a lot of other potentially useful technologies.<br />
<br />
<h5>To Be Continued</h5>The next post will show how to use <a href="http://alexo.github.com/wro4j/">wro4j</a> library to integrate the extended CSS compiler into java project. We picked up wro4j, because it seems to be the most mature, has the most users and offers additional nice features once you have it in your project.<br />
</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com8tag:blogger.com,1999:blog-3355333693246614846.post-30516375461645284432012-06-01T00:53:00.000-07:002012-06-01T00:53:35.663-07:00Stanford Free Crypto Class - Review<div style="text-align: justify;">The free <a href="http://www.crypto-class.org/">online cryptography course</a> I attended this April was taught by Stanford professor <a href="http://en.wikipedia.org/wiki/Dan_Boneh">Dan Boneh</a>. Its infrastructure and organization are provided by <a href="https://www.coursera.org/">coursera</a> company. <br />
<br />
The course was everything I expect from an university course. It explains both the theory and its practical consequences. It shows how ciphers work, how to use them, what are their limitations and why they have been designed the way they have been designed.<a name='more'></a><br />
<br />
It also shows typical mistakes in cipher design and implementation. Attacks are demonstrated on real protocols and systems and much more convincing than standard "Bob changes love letter" scenarios.<br />
<br />
The most important thing we learned is that a secure cipher has an exact definition and how to use that definition to prove a cipher security. We learned also that cipher security definitions have their limits and that one can securely use a cipher only if he understands its limits.<br />
<br />
<h4>Two Parts</h4>The course is divided into two parts and only first one was available. The first part gradually builds symmetric ciphers until they are as secure as currently possible. It also contains an introduction into key exchange protocols and asymmetric cryptography. <br />
<br />
The second part will contain more about protocols, privacy and electronic signatures. It should be released in fall. <br />
<br />
<h4>Prerequisites</h4>The course is mostly self contained. Passive knowledge of high school level probability and discreet math is helpful, but not necessary. A motivated high school student with interest in math should be able to pass the course without problems.<br />
<br />
However, the cryptography is based on math and the course contains a lot of theory. The course would be very challenging for those who do not like learning math. It is much easier for those who studied math, physics, computer science or other related field. <br />
<br />
Of course, if you plan to do also programming assignments, you should have basic understanding of some programming language. <br />
<br />
<h4>Weekly Demands</h4>The part one had 6 weeks. A set of lectures and homework have been released each week. The lecture is a 2-3 hours long video split into 10-20 minutes long segments. The total running time is usually closer to 3 hours. Subtitles and slide show are available too. <br />
<br />
The lecturer assumes that you have the pause and rewind buttons available. He does not make pauses for note taking or thinking, so watching 1 hour log video may easily take more than 1.5 hour. <br />
<br />
The stoppable videos turned out to be surprisingly superior over life lectures. I could could stop for quizzes, thinking or whenever I lost concentration and need a break. I could watch the same segment multiple times. As a result, I remembered and understood all their content after I finished with them. <br />
<br />
The theoretical part of the homework is mandatory while programming assignments are optional. Each theoretical homework has 9-15 questions and is doable within 2-3 hours. With one notable exception, the programming assignment are doable within 0.5-4 hours.<br />
<br />
Note: those are only estimates. I did not measured how much time it really took. <br />
<br />
<h4>Discussion Forum and IRC</h4>The IRC channel and discussion forum were good enough replacement for normal face to face communication. Both have been active right after lectures release and right before the deadline. They were dead silent in between.<br />
<br />
The course staff and the professor participated in discussion forums. They answered more difficult or interesting questions, provided additional learning materials and so on. <br />
<br />
I found joining the IRC channel worth the time it took. It provided additional motivation to do and learn more than required by the course. Some people used advanced algorithms to solve assigned problems, so I learned about those algorithms. Others wrote faster solutions then mine was, so I had motivation to spend additional time to optimize my solution.<br />
<br />
<h4>Complains and Requests</h4>Complains and requests on discussion forums have been taken seriously by the lecturer and the staff. They listened, answered and more importantly have been willing to change certain things. <br />
<br />
For example, programming assignments have been mandatory when the course started. As this was not known up front and the first week assignment was very time consuming, the people complained. Programming assignments are now optional. <br />
<br />
<h4>Organizational Problems</h4>Apparently the work required to put together course like this was bigger than expected. The course was supposed to start in January. It was postponed without promised date and then postponed again. Just when I gave up on it, the course started.<br />
<br />
The lecture of week 4 came after week and half, week 5 took also week and half and the last one took two and half weeks. While the first week and half space helped me to get some breathing space during the Easter, the last two and half weeks long gap was too much.<br />
<br />
I guess this is going to be less of a problem in next releases. The videos, subtitles and whatever else are already done. Updating them should be less work and easier to predict then having to create an entirely new set.<br />
<br />
There was very little communication about those delays. I would prefer to be told about the delay as soon as possible.<br />
<br />
<h4>Beyond the Course</h4>The course provided enough inspiration to get me occupied for the next year. It also referenced a lot of articles about topics related to cryptography. <br />
<br />
Basically, I can come back to it any time I'm bored and immediately find something to do.<br />
<br />
<h4>Conclusion</h4>I would recommend the course for anyone who has interest in cryptography and enough time to complete it. There is nothing to lose and a lot to gain.<br />
<br />
If nothing else, completing the course is a great brain exercise.</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com10tag:blogger.com,1999:blog-3355333693246614846.post-46171819839223662972012-05-01T01:10:00.003-07:002012-05-01T01:14:26.319-07:00Secure Encryption in Java<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Last time I <a href="http://meri-stuff.blogspot.com/2011/12/apache-shiro-part-3-cryptography.html">wrote about cryptography</a>, I outlined Apache Shiro crypto API and shown how to use its two symmetric ciphers. I also wrote that "You do not need more to encrypt and decrypt sensitive data in your applications." <br />
<br />
I <a href="http://www.crypto-class.org/">learned more</a> about cryptography and found out that you need to know more. What I wrote is true to some extend, but unless you are careful your sensitive data may not be secure against all attackers.<br />
<br />
Out of the box Shiro provides Blowfish-CBC and AES-CBC encryption methods and I recommended to use them. Both have been designed to protect against passive eavesdropping attacker and are good at it. Unfortunately, real attackers are more sophisticated and may break the system based on them.<a name='more'></a><br />
<br />
Notice the "may". The attacker can succeed only if attacked system cooperates with him at least little bit. If you want to use these ciphers, you have to know how to write the system securely. Of course, the other option is to use stronger cipher and avoid the problem completely. <br />
<br />
Few needed theoretical <a href="#Theory">terms and concepts</a> are explained in the first chapter. Second chapter shows how to encrypt data in a <a href="#AuthenticatedEncryptionwithShiro">more secure way</a>. Then we describe how Blowfish-CBC and AES-CBC <a href="#CipherBlockChaining(CBC)">work</a> and show two possible <a href="#Attacks">attacks</a> on them. Finally, the end of the post contains references to <a href="#Resources">other resources</a>. <br />
<br />
<a name="TOC"></a><h4>Table of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><ul><li><a href="#Theory">Theory</a></li>
<ul><li><a href="#TheoryActivevsPassiveAttacker">Active vs Passive Attacker</a></li>
<li><a href="#TheoryBlockCiphers">Block Ciphers</a></li>
<li><a href="#TheoryAuthenticatedEncryption">Authenticated Encryption</a></li>
<li><a href="#TheoryVulnerableCiphers">Vulnerable Ciphers</a></li>
<li><a href="#TheorySecureCiphers">Secure Ciphers</a></li>
</ul><li><a href="#AuthenticatedEncryptionwithShiro">Authenticated Encryption with Shiro</a></li>
<ul><li><a href="#AuthenticatedEncryptionwithShiroInstallAuthenticatedOperationModes">Install Authenticated Operation Modes</a></li>
<li><a href="#AuthenticatedEncryptionwithShiroIntegratewithShiro">Integrate with Shiro</a></li>
<li><a href="#AuthenticatedEncryptionwithShiroTestCase">Test Case</a></li>
<li><a href="#AuthenticatedEncryptionwithShiroNoteonJava7">Note on Java 7</a></li>
</ul><li><a href="#CipherBlockChaining(CBC)">Cipher Block Chaining (CBC)</a></li>
<ul><li><a href="#CipherBlockChaining(CBC)Basics">Basics</a></li>
<li><a href="#CipherBlockChaining(CBC)FirstBlockInitializationVector">First Block - Initialization Vector</a></li>
<li><a href="#CipherBlockChaining(CBC)LastBlockPadding">Last Block - Padding</a></li>
</ul><li><a href="#Attacks">Attacks</a></li>
<ul><li><a href="#AttacksDataTampering">Data Tampering</a></li>
<ul><li><a href="#AttacksDataTamperingPotentialDanger">Potential Danger</a></li>
<li><a href="#AttacksDataTamperingTheAttack">The Attack</a></li>
<li><a href="#AttacksDataTamperingTestCase">Test Case</a></li>
</ul><li><a href="#AttacksDecrypttheCipher">Decrypt the Cipher</a></li>
<ul><li><a href="#AttacksDecrypttheCipherPaddingOracle">Padding Oracle</a></li>
<li><a href="#AttacksDecrypttheCipherGeneralIdea">General Idea</a></li>
<li><a href="#AttacksDecrypttheCipherAlgorithm">Algorithm</a></li>
<li><a href="#AttacksDecrypttheCipherTestCase">Test Case</a></li>
</ul></ul><li><a href="#Resources">Resources</a></li>
<li><a href="#End">End</a></li>
</ul></div></div><a name="Theory"></a><h4>Theory</h4>We start with a few theoretical concepts. If you do not want to read it, go directly to the chapter with <a href="#AuthenticatedEncryptionwithShiro">the solution</a>. <br />
<br />
First important thing to describe is the difference between a passive and an active attacker. Then we explain what are block ciphers and what is an authenticated encryption. Last two subchapters list selected vulnerable and selected secure ciphers.<br />
<br />
<a name="TheoryActivevsPassiveAttacker"></a><h5>Active vs Passive Attacker</h5>Eavesdropping only attacker is mostly passive. He is able to read encrypted communication, but is unable to modify it or send new ciphertexts to communicating applications. He is not able to influence the communication, he only listens to it. His passivity has only one exception: the attacker is able to give unencrypted information to one of communicating parties and obtain ciphertext with that exact information.<br />
<br />
Real-world attackers are often more active. They compose their own messages and send them to communicating application. The application then assumes that those messages are encrypted, so it tries to decrypt them. The attacker observes its reaction (returned error code, time needed to answer and so on), and learn more about the cipher.<br />
<br />
If he is lucky, he may use obtained knowledge to break the cipher or to plant false information.<br />
<br />
<a name="TheoryBlockCiphers"></a><h5>Block Ciphers</h5>Block ciphers are able to encrypt only short messages. For example, AES can encrypt only 16 bytes long messages and Blowfish is able to encrypt only 8 bytes long messages.<br />
<br />
Longer messages are split into blocks. Each block is combined with previously encrypted blocks and passed to the block cipher. Block combining is called an operation mode and there are multiple secure ways how to do it. <br />
<br />
Two active attacks discussed in this post are attacks on CBC operation mode. Block ciphers themselves are secure.<br />
<br />
<a name="TheoryAuthenticatedEncryption"></a><h5>Authenticated Encryption</h5>Authenticated encryption rejects any modified ciphertext as invalid. It is not possible to take encrypted data, modify them and end up with valid ciphertext. This property is also called a ciphertext integrity.<br />
<br />
The cipher checks integrity first and rejects all modified messages the same way. As the attacker can not pass through the integrity check, he gains nothing from sending new messages to the application. <br />
<br />
Authentication and ciphertext integrity are usually, but not always, provided by the operation mode.<br />
<br />
<a name="TheoryVulnerableCiphers"></a><h5>Vulnerable Ciphers</h5>Any cipher that does not provide ciphertext integrity or authenticated encryption is probably vulnerable to some active attack. It does not matter which encryption library is used. Encryption algorithms are defined in standards and amount to the same thing regardless of the used library. <br />
<br />
Otherwise said, if it is possible to create valid ciphertext without knowing the secret key, then it is likely that some active attack exists, even if it is not know yet. <br />
<br />
This post describes two attacks CBC operation mode. Once you understand both these attacks and differences between passive and active attacker, you should be able to come up with similar attacks on CFB, CTR, OFB or other non-authenticated cipher modes.<br />
<br />
<a name="TheorySecureCiphers"></a><h5>Secure Ciphers</h5>Authenticated encryptions are secure against active attackers. Most common operation modes that provide also authentication are:<br />
<ul><li>GCM</li>
<li>EAX</li>
<li>CCM</li>
</ul><br />
Replace CBC with one of these modes (e.g. AES-EAX) and you have a cipher secure against an active attacker. <br />
<br />
Of course, secure against an active attacker does not mean that the cipher has no other real-world limitation. For example, most ciphers are safe only if the user changes the key after too much data have been encrypted. If you are serious about data encryption, you should study and know those limitations.<br />
<br />
<a name="AuthenticatedEncryptionwithShiro"></a><h4>Authenticated Encryption with Shiro</h4>This chapter shows how to use an authenticated encryption in Java. For those who skipped the <a href="#Theory">theory</a>, authenticated encryption protects against data tampering. Nobody without the secret key will be able to modify an encrypted message and an active attack on such cipher is impossible.<br />
<br />
Apache Shiro does not implement its own encryption algorithms. It delegates all work to Java Cryptography Extension (JCE) which is available in each Java runtime. Shiro 'only' provides easy to use API and secure defaults. <br />
<br />
Therefore, we have to do two things:<br />
<ul><li>install authenticated operation modes into Java Cryptography Extension,</li>
<li>integrate Shiro with new authenticated operation modes.</li>
</ul><br />
<a name="AuthenticatedEncryptionwithShiroInstallAuthenticatedOperationModes"></a><h5>Install Authenticated Operation Modes</h5>Java Cryptography Extension is an extensible API. All classes and algorithms are created by providers and new provider can be installed into the system at any time. The provider can be either:<br />
<ul><li>installed into java runtime and be available to all java applications,</li>
<li>distributed and initialized together with the application.</li>
</ul><br />
We will show how to add <a href="http://www.bouncycastle.org/">Bouncy Castle</a> into the project. Bouncy Castle is a provider distributed under the MIT license and contains both EAX and GCM operation modes. <br />
<br />
Bouncy Castle distributes multiple jar files, each optimized for different java version. As our demo project uses Java 6, we have to use <code>bcprov-jdk16</code> library. Add maven dependency into pom.xml:<br />
<pre class="brush:xml"><dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk16</artifactId>
<version>1.46</version>
</dependency>
</pre><br />
Once the library is present, you have to install its provider into java security system. Run following method at the application start-up:<br />
<pre class="brush:java">private BouncyCastleProvider installBouncyCastle() {
BouncyCastleProvider provider = new BouncyCastleProvider();
Security.addProvider(provider);
return provider;
}
</pre><br />
Bouncy Castle is now installed and both authenticated operation modes are available.<br />
<br />
<a name="AuthenticatedEncryptionwithShiroIntegratewithShiro"></a><h5>Integrate with Shiro</h5>Shiro cryptography package is basically an easy to use facade over JCE. It configures JCE objects to use secure defaults and adds thread safety to it.<br />
<br />
Extend <code>DefaultBlockCipherService</code> class to take advantage of these features. Most of configuration is done by that class, but we still have to specify three things:<br />
<ul><li>block cipher name and parameters,</li>
<li>operation mode,</li>
<li>padding.</li>
</ul><br />
Block cipher name is specified in constructor parameter. We will use AES, because it requires no additional configuration. Neither GCM nor CCM require padding, so we have to specify the padding <code>NONE</code>.<br />
<br />
New cipher service:<br />
<pre class="brush:java">class GCMCipherService extends DefaultBlockCipherService {
private static final String ALGORITHM_NAME = "AES";
public GCMCipherService() {
super(ALGORITHM_NAME);
setMode(OperationMode.GCM);
setPaddingScheme(PaddingScheme.NONE);
}
}
</pre><br />
That is it. You may use the new authenticating cipher as <a href="http://meri-stuff.blogspot.com/2011/12/apache-shiro-part-3-cryptography.html#CryptoEncryption">any other Shiro cipher service</a>. <br />
<br />
<a name="AuthenticatedEncryptionwithShiroTestCase"></a><h5>Test Case</h5>We created a simple test case to demonstrate that the integrity check works. It encrypts a message and changes third byte of its ciphertext. If the cipher provides an authentication, then an attempt to decrypt modified ciphertext results in runtime exception:<br />
</div><pre class="brush:java">@Test
public void testGCMAuthentication() {
String message = "secret message";
GCMCipherService gcmCipher = new GCMCipherService();
assertIngetrityCheck(message, gcmCipher);
}
private void assertIngetrityCheck(String message, DefaultBlockCipherService cipher) {
byte[] key = cipher.generateNewKey().getEncoded();
byte[] messageBytes = CodecSupport.toBytes(message);
ByteSource encrypt = cipher.encrypt(messageBytes, key);
// change the ciphertext
encrypt.getBytes()[3] = 0;
try {
// it should be impossible to decrypt changed ciphertext
cipher.decrypt(encrypt.getBytes(), key).getBytes();
} catch (Exception ex) {
return;
}
fail("It should not be possible to decrypt changed ciphertext.");
}
</pre><div style="text-align: justify;"><br />
<a name="AuthenticatedEncryptionwithShiroNoteonJava7"></a><h5>Note on Java 7</h5>According to <a href="http://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html">documentation</a>, Java 7 supports two authenticated operation modes: CCM and GCM. Theoretically, you should not need a third party cryptography provider.<br />
<br />
Unfortunately, Oracle could not provide a full implementation of these modes in first JDK 7 release. They would <a href="http://mail.openjdk.java.net/pipermail/security-dev/2011-April/003097.html">like to add it</a> in an update release, so the situation may change in the future. Oracle bug database contains two <a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6996769">related</a> <a href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7031343">bugs</a>. <br />
<br />
Java 1.7.0_01 still does not have them. <br />
<br />
<a name="CipherBlockChaining(CBC)"></a><h4>Cipher Block Chaining (CBC)</h4>The last thing we needed before we can describe promised attacks is cipher block chaining (CBC) operation mode. This operation mode is sufficiently secure against passive attacker, reasonably fast and easy to implement. <br />
<br />
Unfortunately, it also is vulnerable to active attacks.<br />
<br />
<a name="CipherBlockChaining(CBC)Basics"></a><h6>Basics</h6>CBC is used to encrypt a long message with block cipher. Block ciphers are able to encrypt only short blocks of data, so it starts by splitting the message into short blocks. <br />
<br />
First and last blocks are special cases. We will explain what to do with them in following subchapters. For now, assume that the message beginning is already encrypted and its i-th block <code>m<sub>i</sub></code> corresponds to ciphertext <code>c<sub>i</sub></code>.<br />
<br />
Encrypt the next message block in two steps:<br />
<ul><li>xor the block with ciphertext of the previous block (e.g. <code>m<sub>i</sub>⊕c<sub>i-1</sub></code>),</li>
<li>encrypt the result with a block cipher (e.g. <code>Blowfish(key, m<sub>i</sub>⊕c<sub>i-1</sub>)</code>).</li>
</ul><br />
Example: suppose that the secret message has three blocks and we are trying to encrypt it with Blowfish-CBC. The first block is already encrypted and its ciphertext is <code>1, 2, 3, 4, 5, 6, 7, 8</code>. The second block is a byte array <code>1, 0, 1, 0, 1, 0, 1, 0</code>. <br />
<br />
Step 1: xor the first block ciphertext with the second block: <br />
<pre class="brush:text">1, 2, 3, 4, 5, 6, 7, 8 ⊕ 1, 0, 1, 0, 1, 0, 1, 0 = 0, 2, 2, 4, 4, 6, 6, 8
</pre><br />
Step 2: encrypt the result with blowfish algorithm:<br />
<pre class="brush:text">Blowfish(secret_key, {0, 2, 2, 4, 4, 6, 6, 8})
</pre><br />
<a name="CipherBlockChaining(CBC)FirstBlockInitializationVector"></a><h6>First Block - Initialization Vector</h6>First block has no previous block to be combined with. Therefore, we will generate a block of random data called initialization vector. The initialization vector is used as a very first block of data. It is xor-ed with the first message block and the result is encrypted with a block cipher.<br />
<br />
Initialization vector is send unencrypted as a first block of the ciphertext. The recipient would be unable to decrypt the ciphertext without it and there is no reason to keep it secret.<br />
<br />
Example: suppose that the secret message has three blocks and we are trying to encrypt it with Blowfish-CBC. The first block is a byte array <code>1, 1, 1, 1, 1, 1, 1, 1</code>.<br />
<br />
Step 1: generate random initialization vector: <br />
<pre class="brush:text">1, 8, 2, 7, 3, 6, 4, 5
</pre><br />
Step 2: xor the first block with the initialization vector: <br />
<pre class="brush:text">1, 8, 2, 7, 3, 6, 4, 5 ⊕ 1, 1, 1, 1, 1, 1, 1, 1 = 0, 9, 3, 6, 2, 7, 5, 4
</pre><br />
Step 3: encrypt the result with blowfish algorithm:<br />
<pre class="brush:text">Blowfish(secret_key, {0, 9, 3, 6, 2, 7, 5, 4})
</pre><br />
Step 4: combine initialization vector and ciphertext. If the result of Blowfish function in previous step is <code>1, 2, 3, 4, 5, 6, 7, 8</code>, then the ciphertext is:<br />
<pre class="brush:text">1, 8, 2, 7, 3, 6, 4, 5, 1, 2, 3, 4, 5, 6, 7, 8
</pre><br />
<a name="CipherBlockChaining(CBC)LastBlockPadding"></a><h6>Last Block - Padding</h6>Block ciphers are able to encrypt messages of fixed length and last block is often shorter than that. As the cipher is unable to encrypt it, we need a way to add additional bytes to the end of the message. <br />
<br />
Shiro uses PKCS#5 padding by default. Each its byte is equal to the length of the padding: <br />
<ul><li>If the last block is too short, count how many bytes are missing and fill missing bytes with that number. </li>
<li>If the last block has the right size, treat the message as if it would be missing whole block. Add a new padding block to it. Each its byte will be equal to the block size.</li>
</ul><br />
Example 1: suppose that the secret message has three blocks and we are trying to encrypt it with Blowfish-CBC. The last block is byte array <code>1, 1, 1, 1</code>. Padded block: <br />
<pre class="brush:text">1, 1, 1, 1, 4, 4, 4, 4
</pre><br />
Example 2: suppose that the secret message has three blocks and we are trying to encrypt it with Blowfish-CBC. The last block is byte array <code>8, 7, 6, 5, 4, 3, 2, 1</code>. Last block and padding: <br />
<pre class="brush:text">8, 7, 6, 5, 4, 3, 2, 1, 8, 8, 8, 8, 8, 8, 8, 8
</pre><br />
<a name="Attacks"></a><h4>Attacks</h4>Finally, we are ready to show two different attacks on CBC based ciphers and prove that the problem is real. Our <a href="https://github.com/SomMeri/org.meri.crypto.demo">sample project</a> contains tests cases for both attacks on both AES-CBC and Blowfish-CBC ciphers. <br />
<br />
First subchapter shows how to change the beginning of encrypted text to any message of our choice. Second subchapter explains how to decrypt the ciphertext.<br />
<br />
<a name="AttacksDataTampering"></a><h5>Data Tampering</h5>Data tampering attack is possible only if the attacker already <a href="#AttacksDecrypttheCipher">knows the content</a> of encrypted message. Such attacker can change first message block to whatever he wishes to.<br />
<br />
In particular, it is possible to: <br />
<ul><li>change first 16 bytes of AES-CBC encrypted message,</li>
<li>change first 8 bytes of Blowfish-CBC encrypted message.</li>
</ul><br />
<a name="AttacksDataTamperingPotentialDanger"></a><h6>Potential Danger</h6>Whether this type of attack is dangerous depends a lot on circumstances. <br />
<br />
If you use the cipher to send password through network, then data tampering is not so dangerous. At worst, a legitimate user will get login denied. Similarly, if your encrypted data are stored on some read-only storage, then you do not have to worry about data tampering. <br />
<br />
However, if you are sending bank order through the network, data tampering is a real threat. If someone changes the message <code>Pay Mark 100$</code> into <code>Pay Tom 9999$</code>, Tom will get 9999$ he should not get.<br />
<br />
<a name="AttacksDataTamperingTheAttack"></a><h6>The Attack</h6>Encrypted message has three parts:<br />
<ul><li>random initial vector,</li>
<li>first block of ciphertext,</li>
<li>the rest of the message.</li>
</ul><br />
The recipient decrypts the first block of ciphertext with the block cipher. This gives him <code>message block⊕initial vector</code>. To get the message, he has to xor this value with the initial vector:<br />
<pre class="brush:text">message block ⊕ initial vector ⊕ initial vector = message block</pre><br />
The initial vector is transferred together with the message and an active attacker can change it. If the attacker replaces the original initial vector with <code>another iv</code>, then the recipient will decrypt another message:<br />
<pre class="brush:text">message block ⊕ initial vector ⊕ another iv = another message</pre><br />
Rearrange the previous equation and you get:<br />
<pre class="brush:text">another iv = message block ⊕ initial vector ⊕ another message</pre><br />
If the attacker knows both initial vector and content of the encrypted message, then he can modify the message to anything he wants. All he has to do is to xor the original initial vector with both known message content and desired message.<br />
<br />
Whoever decrypts the modified ciphertext will obtain a modified message instead of the original one.<br />
<br />
<a name="AttacksDataTamperingTestCase"></a><h6>Test Case</h6>We created a simple test case that demonstrate this attack on a message encrypted with AES-CBC. <br />
<br />
Imagine that an attacker captured an encrypted email and somehow knows what is in it:</div><pre class="brush:java">//original message
private static final String EMAIL = "Hi,\n" +
"send Martin all requested money please.\n\n" +
"With Regards, \n" +
"Accounting\n";
</pre><div style="text-align: justify;"><br />
The attacker can change only first 16 bytes of the message, so he decides to redirect the money to Andrea:</div><pre class="brush:java">//changed message
private static final String MODIFICATION = "Hi,\n" +
"give Andrea all requested money please.\n\n" +
"With Regards, \n" +
"Accounting\n";
</pre><div style="text-align: justify;"><br />
Following test case encrypts the message and modifies the ciphertext. Modified ciphertext is decrypted and compared to the expected message:</div><pre class="brush:java">@Test
public void testModifiedMessage_AES() {
//create cipher and the secret key
StringCipherService cipher =
new StringCipherService(new AesCipherService());
byte[] key = cipher.generateNewKey();
//encrypt the message
byte[] ciphertext = cipher.encrypt(EMAIL, key);
//attack: modify the encrypted message
for (int i = 0; i < 16; i++) {
ciphertext[i] = (byte)(ciphertext[i] ^
MODIFICATION.getBytes()[i] ^
EMAIL.getBytes()[i]);
}
//decrypt and verify
String result = cipher.decrypt(ciphertext, key);
assertEquals(MODIFICATION, result);
}</pre><div style="text-align: justify;"><br />
Of course, similar attack can be done on Blowfish-CBC. We can change only first 8 bytes this time:</div><pre class="brush:java">@Test
public void testModifiedMessage_Blowfish() {
String email = "Pay 100 dollars to them, but nothing more. Accounting\n";
StringCipherService cipher =
new StringCipherService(new BlowfishCipherService());
byte[] key = cipher.generateNewKey();
byte[] ciphertext = cipher.encrypt(email, key);
String modified = "Pay 900 dollars to them, but nothing more. Accounting\n";
for (int i = 0; i < 8; i++) {
ciphertext[i] = (byte)(ciphertext[i] ^
modified.getBytes()[i] ^
email.getBytes()[i]);
}
String result = cipher.decrypt(ciphertext, key);
assertEquals(modified, result);
}</pre><div style="text-align: justify;"><br />
<a name="AttacksDecrypttheCipher"></a><h5>Decrypt the Cipher</h5>The second attack allows an attacker to decrypt the secret message. The attack is possible only if the application that decrypts secret messages cooperates with the attacker.<br />
<br />
<a name="AttacksDecrypttheCipherPaddingOracle"></a><h6>Padding Oracle</h6>The attacker creates a lot of fake ciphertexts and sends them to the recipient. As he tries to decrypt those messages, one of these things will happen: <br />
<ul><li>the ciphertext decrypts to meaningless garbage,</li>
<li>modified message will not be valid ciphertext at all.</li>
</ul><br />
If the application behaves the same way in both cases, then everything is ok. If it behaves differently, then the attacker is able to decrypt the ciphertext. There is only one way how the CBC based ciphertext can be incorrect - if the padding is wrong. <br />
<br />
For example, if ciphertext contains an encrypted password, the vulnerable server may respond with "login denied" in case of wrong decrypted password and with runtime exception in case of invalid ciphertext. If this is the case, then the attacker can recover the password.<br />
<br />
<a name="AttacksDecrypttheCipherGeneralIdea"></a><h6>General Idea</h6>Each fake message has two parts: a fake initial vector and one message block. Both are sent to the server. If it answers "padding right", then we know that:<br />
<pre class="brush:text">message ⊕ original iv ⊕ fake iv = valid padding
</pre><br />
The only unknown variable in the above equation is the message. The <code>original iv</code> is previous ciphertext block, <code>fake iv</code> was created by us and the valid padding is one of <code>1</code> or <code>2, 2</code> or <code>3, 3, 3</code> or <code>...</code> or <code>8, 8, ..., 8</code> and so on. <br />
<br />
Therefore, we can calculate the block content as:<pre class="brush:text">message = valid padding ⊕ original iv ⊕ fake iv
</pre><br />
<a name="AttacksDecrypttheCipherAlgorithm"></a><h6>Algorithm</h6>Start by recovering the last block byte. Each fake initial vectors starts with a lot of 0 and ends with a different last byte. This way, we can be almost sure that the server answers "padding right" only on a message that ends with 1. Use the previous chapter equation to calculate the last block byte.<br />
<br />
Getting the second last byte of the message is very similar. The only difference is that we have to craft a ciphertext that decrypts into the second shortest padding <code>2, 2</code>. The last byte of the message is already known, so enforcing 2 as the last value is easy. The beginning of the initial vector is unimportant and set that to 0. <br />
<br />
Then we try all possible values for the second last byte of the initial vector. Once the server answers "padding right", we can get the second last message byte from the same formula as before: <code>original iv ⊕ fake iv ⊕ 2</code>.<br />
<br />
We calculate third last message byte out of fake message with padding <code>3, 3, 3</code>; fourth out of message with padding <code>4, 4, 4, 4</code>; and so on until the whole block is decrypted. <br />
<br />
<a name="AttacksDecrypttheCipherTestCase"></a><h6>Test Case</h6>The vulnerable server is simulated with a <code>PaddingOraculum</code> class. Each instance of this class generates a random secret key and keeps it private. It exposes only two public methods:<br />
<ul><li><code>byte[] encrypt(String message)</code> - encrypts a string with secret key,</li>
<li><code>boolean verifyPadding(byte[] ciphertext)</code> - returns whether the padding is right.</li>
</ul><br />
The padding oraculum attack is implemented in <code>decryptLastBlock</code> method. The method decrypts last block of encrypted message:</div><pre class="brush:java">private String decryptLastBlock(PaddingOraculum oraculum, byte[] ciphertext) {
// extract relevant part of the ciphertext
byte[] ivAndBlock = getLastTwoBlocks(ciphertext,
oraculum.getBlockSize());
// modified initial vector
byte[] ivMod = new byte[oraculum.getBlockSize()];
Arrays.fill(ivMod, (byte) 0);
// Start with last byte of the last block and
// continue to the first byte.
for (int i = oraculum.getBlockSize()-1; i >= 0; i--) {
// add padding to the initial vector
int expectedPadding = oraculum.getBlockSize() - i;
xorPad(ivMod, expectedPadding);
// loop through possible values of ivModification[i]
for (ivMod[i] = -128; ivMod[i] < 127; ivMod[i]++) {
// create fake message and verify its padding
byte[] modifiedCiphertext = replaceBeginning(ivAndBlock, ivMod);
if (oraculum.verifyPadding(modifiedCiphertext)) {
// we can stop looping
// the ivModification[i] =
// = solution ^ expectedPadding ^ ivAndBlock[i]
break;
}
}
// remove the padding from the initial vector
xorPad(ivMod, expectedPadding);
}
// modified initial vector now contains the solution xor
// original initial vector
String result = "";
for (int i = 0; i < ivMod.length; i++) {
ivMod[i] = (byte) (ivMod[i] ^ ivAndBlock[i]);
result += (char) ivMod[i];
}
return result;
}
</pre><div style="text-align: justify;"><br />
Our sample project contains two <a href="https://github.com/SomMeri/org.meri.crypto.demo/blob/master/src/test/java/org/meri/crypto/demo/cbcattacks/AttacksDemo.java#L66">test cases</a>. One encrypts message with AES-CBC and then uses the padding oraculum to the last block of the ciphertext. The other do the same thing with Blowfish-CBC.<br />
<br />
Decrypt Blowfish-CBC test case:<br />
</div><pre class="brush:java">@Test
public void testPaddingOracle_Blowfish() {
String message = "secret message!";
PaddingOraculum oraculum = new PaddingOraculum(
new BlowfishCipherService());
//Oraculum encrypts the message with a secret key.
byte[] ciphertext = oraculum.encrypt(message);
//use oraculum to decrypt the message
String result = decryptLastBlock(oraculum, ciphertext);
//the original message had padding 1
assertEquals("essage!"+(char)1, result);
}
</pre><div style="text-align: justify;"><br />
<a name="Resources"></a><h4>Resources</h4>Additional related resources: <br />
<ul><li>Free online Stanford <a href="http://www.crypto-class.org/">crypto class</a>.</li>
<li>The original paper with <a href="http://www.iacr.org/archive/eurocrypt2002/23320530/cbc02_e02d.pdf">padding oracle attack</a> from 2002.</li>
<li>A good <a href="http://www.limited-entropy.com/padding-oracle-attacks">alternative explanation</a> of padding oracle attack.</li>
<li>A <a href="http://www.educatedguesswork.org/2011/09/security_impact_of_the_rizzodu.html">beast attack on CBC</a> explained. It is based on padding oracle attack.</li>
<li>Stack exchange <a href="http://security.stackexchange.com/questions/2202/lessons-learned-and-misconceptions-regarding-encryption-and-cryptology">thread</a> on common cryptography mistakes.</li>
</ul><br />
<a name="End"></a><h4>End</h4>No cipher provides absolute safety against all possible attacks. Instead, they provide protection only against well defined classes of attacks. Ciphers are secure only as long as the potential threat to the system matches the cipher strength.<br />
<br />
Protection against an active attack can be done in two ways:<br />
<ul><li>Make an active attack impossible by design.</li>
<li>Use an authenticated encryption.</li>
</ul><br />
Using an authenticated encryption is arguably easier and should be the preferred option. Ruling out the active attacker is error prone and more risky. <br />
<br />
All code sample used in this post are available <a href="https://github.com/SomMeri/org.meri.crypto.demo">on Github</a>. <br />
</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com12tag:blogger.com,1999:blog-3355333693246614846.post-25020042057363741192012-04-01T00:07:00.000-07:002012-04-01T00:07:29.360-07:00Writing Eclipse Plugins Tutorial - Part 1<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Eclipse is one of three most popular java development IDEs. One reason for its success is its extensibility. Writing eclipse plugins can be very fast and easy for anyone who knows what to do and already done it.<br />
<br />
Unfortunately, doing something in Eclipse for the first time can be very time consuming and frustrating. Eclipse framework is huge, powerful and sometimes complicated. It may be difficult to figure out which features are available and how to use them. <br />
<br />
This tutorial explains basics of all eclipse features needed to automate a simple java refactoring task. It shows how to add new items into the menu and how to analyze, modify and format java source code. It also shows how to use dialogs to communicate with users.<a name='more'></a><br />
<br />
It is split into two parts. This post explains all needed theory. By the end of this part, you will already know enough about Eclipse framework to be able to finish the plugin with just a little searching. Practically speaking, we will create a plugin that adds new items into the menu and collects all information needed to perform the refactoring. <br />
<br />
Next part of this tutorial will show how to create dialogs and how to modify java sources from the plugin. It was not released yet. <br />
<br />
<a name="TableofContents"></a><h4>Table of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><ul><li><a href="#SamplePlugin">Sample Plugin</a></li>
<li><a href="#IDE">IDE</a></li>
<li><a href="#TargetPlatform">Target Platform</a></li>
<ul><li><a href="#TargetPlatformEasiestTargetPlatform">Easiest Target Platform</a></li>
<li><a href="#TargetPlatformAdvancedTargetPlatform">Advanced Target Platform</a></li>
<ul><li><a href="#TargetPlatformAdvancedTargetPlatformCreateTargetPlatform">Create Target Platform</a></li>
<li><a href="#TargetPlatformAdvancedTargetPlatformConfigureTargetPlatform">Configure Target Platform</a></li>
</ul></ul><li><a href="#SimplePluginProject">Simple Plugin Project</a></li>
<ul><li><a href="#SimplePluginProjectCreatePluginProject">Create Plugin Project</a></li>
<li><a href="#SimplePluginProjectDebugThePlugin">Debug The Plugin</a></li>
<li><a href="#SimplePluginProjectCleanthePlugin">Clean the Plugin</a></li>
</ul><li><a href="#AdapterPattern">Adapter Pattern</a></li>
<ul><li><a href="#AdapterPatternAdapter">Adapter</a></li>
<li><a href="#AdapterPatternAdaptableObjects">Adaptable Objects</a></li>
<ul><li><a href="#AdapterPatternAdaptableObjectsEclipseImplementation">Eclipse Implementation</a></li>
<li><a href="#AdapterPatternAdaptableObjectsWhenToUseIt">When To Use It</a></li>
<li><a href="#AdapterPatternAdaptableObjectsLimitations">Limitations</a></li>
</ul><li><a href="#AdapterPatternThirdPartyAdapter">Third Party Adapter</a></li>
<ul><li><a href="#AdapterPatternThirdPartyAdapterEclipseImplementation">Eclipse Implementation</a></li>
<li><a href="#AdapterPatternThirdPartyAdapterWhenToUseIt">When To Use It</a></li>
<li><a href="#AdapterPatternThirdPartyAdapterLimitations">Limitations</a></li>
</ul><li><a href="#AdapterPatternAdaptanObject">Adapt an Object</a></li>
</ul><li><a href="#AbstractSyntaxTreevsJavaModel">Abstract Syntax Tree vs Java Model</a></li>
<ul><li><a href="#AbstractSyntaxTreevsJavaModelPluginDependency">Plugin Dependency</a></li>
<li><a href="#AbstractSyntaxTreevsJavaModelJavaModel">Java Model</a></li>
<li><a href="#AbstractSyntaxTreevsJavaModelAbstractSyntaxTree">Abstract Syntax Tree</a></li>
</ul><li><a href="#UserInterface">User Interface</a></li>
<ul><li><a href="#UserInterfaceParts,ViewsandEditors">Parts, Views and Editors</a></li>
<ul><li><a href="#UserInterfaceParts,ViewsandEditorsCommonProperties">Common Properties</a></li>
<li><a href="#UserInterfaceParts,ViewsandEditorsEditors">Editors</a></li>
<li><a href="#UserInterfaceParts,ViewsandEditorsViews">Views</a></li>
</ul><li><a href="#UserInterfaceSelection">Selection</a></li>
<ul><li><a href="#UserInterfaceSelectionSelectedObjects">Selected Objects</a></li>
</ul></ul><li><a href="#EclipseMenus">Eclipse Menus</a></li>
<ul><li><a href="#EclipseMenusMenusTypes">Menus Types</a></li>
<li><a href="#EclipseMenusSourceMenu">Source Menu</a></li>
<li><a href="#EclipseMenusMenuFrameworks">Menu Frameworks</a></li>
</ul><li><a href="#TheCommandFramework">The Command Framework</a></li>
<ul><li><a href="#TheCommandFrameworkOverview">Overview</a></li>
<li><a href="#TheCommandFrameworkMenuId">Menu Id</a></li>
<ul><li><a href="#TheCommandFrameworkMenuIdTopLevel">Top Level</a></li>
<li><a href="#TheCommandFrameworkMenuIdSubmenuWithPluginSpy">Sub-menu With Plugin Spy</a></li>
<li><a href="#TheCommandFrameworkMenuIdTheOtherWay">The Other Way</a></li>
<li><a href="#TheCommandFrameworkMenuIdPlanB">Plan B</a></li>
</ul><li><a href="#TheCommandFrameworkLocationURI">Location URI</a></li>
<li><a href="#TheCommandFrameworkCommand">Command</a></li>
<li><a href="#TheCommandFrameworkMenuContribution">Menu Contribution</a></li>
<ul><li><a href="#TheCommandFrameworkMenuContributionMenuContribution">Menu Contribution</a></li>
<li><a href="#TheCommandFrameworkMenuContributionCommand">Command</a></li>
<li><a href="#TheCommandFrameworkMenuContributionSeparator">Separator</a></li>
</ul><li><a href="#TheCommandFrameworkCommandHandler">Command Handler</a></li>
<ul><li><a href="#TheCommandFrameworkCommandHandlerSimplestCommandHandler">Simplest Command Handler</a></li>
<li><a href="#TheCommandFrameworkCommandHandlerBasicConfiguration">Basic Configuration</a></li>
<li><a href="#TheCommandFrameworkCommandHandlerImplementation">Implementation</a></li>
</ul><li><a href="#TheCommandFrameworkEnabling,DisablingandVisibility">Enabling, Disabling and Visibility</a></li>
<ul><li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityVisiblevsHidden">Visible vs Hidden</a></li>
<li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityExpressionLanguage">Expression Language</a></li>
<li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityTestCurrentEditor">Test Current Editor</a></li>
<li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityTestSelectedObjects">Test Selected Objects</a></li>
<li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityCheckNonNullParametersEnablement">Check Non-Null Parameters Enablement</a></li>
<li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityGenerateCustomtoStringEnablement">Generate Custom toString Enablement</a></li>
<li><a href="#TheCommandFrameworkEnabling,DisablingandVisibilityExpressionLanguageAdditionalResources">Expression Language - Additional Resources</a></li>
</ul><li><a href="#TheCommandFrameworkCommandFrameworkAdditionalResources">Command Framework - Additional Resources</a></li>
</ul><li><a href="#TheActionsFramework">The Actions Framework</a></li>
<ul><li><a href="#TheActionsFrameworkOverview">Overview</a></li>
<li><a href="#TheActionsFrameworkImplementingActionDelegate">Implementing Action Delegate</a></li>
<ul><li><a href="#TheActionsFrameworkImplementingActionDelegateCommon">Common</a></li>
<li><a href="#TheActionsFrameworkImplementingActionDelegateMenuTypesandSubInterfaces">Menu Types and Sub-Interfaces</a></li>
<li><a href="#TheActionsFrameworkImplementingActionDelegateSelection">Selection</a></li>
<li><a href="#TheActionsFrameworkImplementingActionDelegateImplementation">Implementation</a></li>
</ul><li><a href="#TheActionsFrameworkMenubarPathandToolbarPath">MenubarPath and ToolbarPath</a></li>
<li><a href="#TheActionsFrameworkConfiguringanAction">Configuring an Action</a></li>
<li><a href="#TheActionsFrameworkAdditionalResources">Additional Resources</a></li>
</ul><li><a href="#UsefulResources">Useful Resources</a></li>
<li><a href="#ToBeContinued">To Be Continued</a></li>
</ul></div></div></div><a name="SamplePlugin"></a><h4>Sample Plugin</h4>We create sample plugin with two features: <br />
<ol><li>check non-null parameters - modifies selected method to check whether its parameters are not <code>null</code>,</li>
<li>custom generate toString - adds <code>toString</code> method to selected class.</li>
</ol><br />
Both features will add a new item into the source menu. Check non-null parameters item will be enabled only if user selects a java method. The feature shows a dialog which allows user to choose a subset of method arguments. Selected method is then modified to check whether chosen arguments are null:<br />
<pre class="brush:java">if (arg1==null || arg2==null || ... || argn==null)
throw new IllegalStateException("The parameter may not be null.");
</pre><br />
Custom generate toString item will be enabled only if user selects a java class. It will show a dialog with list of all its properties. The user selects which properties belongs to toString method. If user selected less than four properties, the feature adds following code to the class:<br />
<pre class="brush:java">@Override
public String toString() {
StringBuilder builder = new StringBuilder(this.getClass()
.getSimpleName());
builder.append(" [ ");
builder.append(b).append(", ").append(c).append(" ]");
return builder.toString();
}
</pre><br />
If user selected less four of more properties, the feature adds following code to the class:<br />
<pre class="brush:java">public String toString() {
StringBuilder builder = new StringBuilder(this.getClass()
.getSimpleName());
builder.append(" [ \n");
builder.append(" a" + ": ").append(a).append("\n");
builder.append(" b" + ": ").append(b).append("\n");
builder.append(" c" + ": ").append(c).append("\n");
builder.append(" d" + ": ").append(d).append("\n");
builder.append(" ]");
return builder.toString();
}
</pre><br />
That is all. The plugin is available <a href="https://github.com/SomMeri/org.meri.eclipse.tutorial.defensiveapitools">on Github</a>.<br />
<br />
<a name="IDE"></a><h4>IDE</h4><div style="text-align: justify;">Each Eclipse release comes in multiple flavors. The version best suitable for plugin writers is called 'Eclipse for RCP and RAP Developers'. The RCP stands for 'rich client platform' which is just another name for Eclipse platform.<br />
<br />
Download and install 'Eclipse for RCP and RAP Developers' from the <a href="http://www.eclipse.org/downloads/">download page</a>. <br />
<br />
<a name="TargetPlatform"></a><h4>Target Platform</h4>First thing to do is to configure target platform. Target platform is another instance of eclipse. It represents the minimal configuration your plugin will work with. Your plugin will be compiled against the target platform. It will also be installed into it and will run inside it whenever you will want to test it. <br />
<br />
The whole workspace can have only one active target platform. It is not project specific, although it would make more sense.<br />
<br />
<a name="TargetPlatformEasiestTargetPlatform"></a><h5>Easiest Target Platform</h5>The easiest is to develop for the same eclipse version as you are running on. This is the default option. If this is the case, all you have to do is to add eclipse SDK into it.<br />
<br />
Install Eclipse SDK:<br />
<ul><li>Go to 'Help' -> 'Install New Software...'.</li>
<li>Choose your eclipse update site, in our case the 'The Eclipse Project Updates' <code>http://download.eclipse.org/eclipse/updates/3.</code> update site.</li>
<li>Check both Eclipse SDK and Eclipse Platform SDK.</li>
<li>Click next, accept license and finish the installation.</li>
</ul><br />
That is it. Nothing else is necessary, you are ready to create <a href="#SimplePluginProject">first plugin project</a>.<br />
<br />
<a name="TargetPlatformAdvancedTargetPlatform"></a><h5>Advanced Target Platform</h5>It is possible to download and maintain the target platform separately. Use this option if you want to be compatible with an older release or if you want to have greater control over the target platform configuration.<br />
<br />
If you are not interested, skip to the next chapter.<br />
<br />
<a name="TargetPlatformAdvancedTargetPlatformCreateTargetPlatform"></a><h6>Create Target Platform</h6>Find and download SDK of whatever you assume your users will have. You may use also a 'regular' version if you can not find the SDK. However, if you download SDK, you will have source code and javadocs available.<br />
<br />
For example, our plugin requires Eclipse Indigo release. The version number of Eclipse Indigo is 3.7, therefore we have to download <a href="http://download.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/index.php">Eclipse 3.7 SDK</a>. The complete list of even older Eclipse SDKs versions is available on <a href="http://archive.eclipse.org/eclipse/downloads/">archived releases</a> page.<br />
<br />
Our plugin will depend only on the eclipse itself, so all we have to do is to unpack downloaded SDK somewhere. If it would require additional plugins, we would have to hunt and download their SDKs too. We would also unpack them and copy into the same directory as eclipse SDK.<br />
<br />
<a name="TargetPlatformAdvancedTargetPlatformConfigureTargetPlatform"></a><h6>Configure Target Platform</h6>Now, we have to configure our RCP eclipse to use prepared target platform.<br />
<br />
Define and activate the target platform:<br />
<ul><li>Go to 'Window' -> 'Preferences' -> 'Plug-in development' -> 'Target Platform'.</li>
<li>Click Add.</li>
<li>Choose 'Nothing: Start with an empty target definition' and click next.</li>
<li>Fill in target name.</li>
<li>Click Add, choose directory, browse to the unpacked eclipse SDK and finish.</li>
<li>Check new target platform as 'Active'.</li>
</ul><br />
Finally, go to 'Plug-in Development' preference page and check 'Include all plug-ins from target in Java search'.<br />
<br />
<a name="SimplePluginProject"></a><h4>Simple Plugin Project</h4>This chapter shows how to create a simple plugin project and how to debug it. The simple plugin does nothing useful. It is only able to show a message to prove that it exist.<br />
<br />
In the end of this chapter, we will remove the sample message and end up with an empty plugin skeleton.<br />
<br />
<a name="SimplePluginProjectCreatePluginProject"></a><h5>Create Plugin Project</h5>We will use Eclipse wizard to generate the plugin. Invoke the wizard from the package explorer:<br />
<ul><li>right click in the package explorer,</li>
<li>choose 'New' -> 'Other..'.,</li>
<li>select 'Plug-in Project' and click Next.</li>
</ul><br />
The wizard has multiple pages. Configure the project name and target platform on the first one. You can use any project name as you want, but the convention is to name the project after the root java package. For example, as we want to put all classes into the <code>org.meri.eclipse.defensiveapitools</code> package, our project name will be org.meri.eclipse.defensiveapitools<br />
<br />
The target platform field contains a version of Eclipse you are developing for. If someone wants to use your plugin, he will be required to download Eclipse with equal or bigger number. An old incompatible Eclipse will refuse to load it. We are fine with developing for current Eclipse, so we will choose 3.7. <br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirvQDXRxsijn1pORTg9tBjQfU8HotLWdl25hXkVUu-Ja5nds4xtPLNGjTSePEzyNbMSvNK3nzMuSw9NvI51GyEGyrkqUCE7YQARdcM_RRMWCptJbcJzsKHAywLdz2OVjBHBo62nqwQ7Nub/s1600/CreatePlugin-1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirvQDXRxsijn1pORTg9tBjQfU8HotLWdl25hXkVUu-Ja5nds4xtPLNGjTSePEzyNbMSvNK3nzMuSw9NvI51GyEGyrkqUCE7YQARdcM_RRMWCptJbcJzsKHAywLdz2OVjBHBo62nqwQ7Nub/s320/CreatePlugin-1.png" /></a></div><br />
Click 'Next'.<br />
<br />
Second page contains basic project information. Fill in the id, plugin version, name and provider however you like. The first important parameter is the execution environment. The plugin will run only on the specified or never java. An eclipse running on older JVM will simply ignore it. We have chosen J2SE-1.6.<br />
<br />
Verify that: <br />
<ul><li>check-box 'Generate an activator, a Java ...' is checked,</li>
<li>check-box 'This plugin will make contributions to the UI' is checked,</li>
<li>the answer to 'Would you like to create a rich client platform application' is no.</li>
</ul><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS8Lruj8umu5MVwZ4xOmJVeKIKBRiqbzk1QOoibXBE3QG8Wp3W-gI-1SxGiggNx0Ajt0gGI9pXOgfqhPwlo53kT1I1Fj6PdPqewJTeXVhi9eTqXoAgqcoy-ZSCQ_bcJFIvlISPoTbZk6i4/s1600/CreatePlugin-2.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS8Lruj8umu5MVwZ4xOmJVeKIKBRiqbzk1QOoibXBE3QG8Wp3W-gI-1SxGiggNx0Ajt0gGI9pXOgfqhPwlo53kT1I1Fj6PdPqewJTeXVhi9eTqXoAgqcoy-ZSCQ_bcJFIvlISPoTbZk6i4/s320/CreatePlugin-2.png" /></a></div><br />
Click 'Next'.<br />
<br />
Choose 'Hello, World Command' template and click 'Finish'. This will create new plugin with sample menu item.<br />
<br />
<a name="SimplePluginProjectDebugThePlugin"></a><h5>Debug The Plugin</h5>The plugin works only inside a running Eclipse. Eclipse supports both manual and automatic JUnit testing. In both cases, the plugin is installed and run inside the target platform configured in the <a href="#TargetPlatform">first chapter</a>. <br />
<br />
This chapter shows only how to do the manual testing. Refer to <a href="http://eclipser-blog.blogspot.com/2007/04/when-you-write-eclipse-plug-ins-sooner.html">other resources</a> to learn how to write junit tests for eclipse plugins.<br />
<br />
Install and run the plugin:<br />
<ul><li>right click on the plugin project,</li>
<li>click 'Debug As' -> 'Eclipse Application'.</li>
</ul><br />
The system will start new eclipse instance. Its main menu has a new entry called 'Sample Menu'. <br />
<br />
Now, you have two running eclipse instances. One is for development and another is for testing. The testing eclipse runs inside the development one. All debugging tools are available. You can place breakpoints, inspect variables and so on:<br />
<ul><li>open generated <code>SampleHandler</code> class in the development eclipse,</li>
<li>put breakpoint inside the <code>execute</code> method,</li>
<li>go back to the test eclipse,</li>
<li>choose 'Sample Menu' and 'Sample Command'</li>
</ul><br />
The execution will stop on the new breakpoint.<br />
<br />
<a name="SimplePluginProjectCleanthePlugin"></a><h5>Clean the Plugin</h5>We have seen that the plugin works, so we can remove generated sample menu from it. <br />
<br />
Remove sample menu item and its handler:<br />
<ul><li>Open plugin.xml file, go to extensions tab and remove all extensions.</li>
<li>Locate and delete generated <code>SampleHandler</code> class.</li>
</ul><br />
The <code>plugin</code> tag inside the plugin.xml should be empty now. Open plugin.xml file and go to the plugin.xml tab:<br />
<pre class="brush:java"><?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
</plugin>
</pre><br />
<a name="AdapterPattern"></a><h4>Adapter Pattern</h4>Eclipse is a huge framework and there is an infinite amount of plugins you can install into it. This creates a huge system of features and plugins that have to cooperate with each other. To keep this system as decoupled and as extensible as possible, Eclipse framework uses <a href="http://en.wikipedia.org/wiki/Adapter_pattern">adapter design pattern</a>. This pattern is so common that you will probably run into it no matter what plugin you will write.<br />
<br />
This design pattern converts a class or interface into another class or interface. The class that performs the conversion is called an adapter. The pattern has two different flavors and eclipse framework supports both. The difference between those two flavors is in who creates the adapter. It can be done either directly by the object to be converted or by an independent adapter factory. <br />
<br />
First sub-chapter shows how to write an adapter. Second sub-chapter is about objects able to create their own adapters and third is about independent adapter factories. Last sub-chapter puts everything together and shows how to convert an object of unknown type.<br />
<br />
<a name="AdapterPatternAdapter"></a><h5>Adapter</h5>Adapter is an object that converts one type into another type. It must represent the other type e.g., if it converts objects to a class, then it must extend that class. If it converts them to an interface, then it must implement that interface. <br />
<br />
Typically, an adapter converts object that has all needed functionality, but does not have the right API. Otherwise said, typical adapter contains very little logic. It only wraps the original object and delegates all work to it. <br />
<br />
Following adapter is able to convert an implementation of <code>Minus</code> interface into the <code>Plus</code> interface:<br />
<pre class="brush:java">public class MinusToPlusAdapter implements Plus {
private final Minus adaptee;
public MinusToPlusAdapter(Minus minus) {
super();
this.adaptee = minus;
}
@Override
public int plus(int x, int y) {
return adaptee.minus(x, -y);
}
}
</pre><br />
<a name="AdapterPatternAdaptableObjects"></a><h5>Adaptable Objects</h5>In a simpler version of this design pattern, the object to be converted creates its own adapters. <br />
<br />
We will show how to create adaptable objects compatible with eclipse framework. The rest of this sub-chapter lists advantages and disadvantages of this adapter pattern flavor.<br />
<br />
<a name="AdapterPatternAdaptableObjectsEclipseImplementation"></a><h6>Eclipse Implementation</h6>An adaptable object must implement <code>IAdaptable</code> interface. The interface has only one method <code>getAdapter(Class type)</code>. It returns either an adapter to the requested type or a <code>null</code>.<br />
<br />
An adaptable object:<br />
<pre class="brush:java">public class MinusImpl implements Minus, IAdaptable {
public int minus(int x, int y) {
return x - y;
}
public Object getAdapter(Class type) {
if (Plus.class.equals(type))
return new MinusToPlusAdapter(this);
return null;
}
}
</pre><br />
The <code>getAdapter</code> method may or may not call the <a href="#AdapterPatternThirdPartyAdapter">global adapter manager</a> to create an adapter. Some Eclipse objects call it, some do not.<br />
<br />
<a name="AdapterPatternAdaptableObjectsWhenToUseIt"></a><h6>When To Use It</h6>Adaptable objects are easy to use and debug. The pattern helps to keep class and interface hierarchies clean. It also provides some decoupling between converted type and required type.<br />
<br />
Use it when: <br />
<ul><li>Converted type already implements too many interfaces.</li>
<li>Converted type have to be compatible with two different features and each requires it to extend another class.</li>
<li>You want to keep the converted type and required interface/class separated.</li>
<li>The adapter needs access to private fields of methods.</li>
</ul><br />
<a name="AdapterPatternAdaptableObjectsLimitations"></a><h6>Limitations</h6>Plugin that relies only on this version of the pattern is not extensible. Third party plugins are not going to be able to add new adapters into it.<br />
<br />
<a name="AdapterPatternThirdPartyAdapter"></a><h5>Third Party Adapter</h5>Second version of this pattern uses an independent factory to create adapters. <br />
<br />
We will show how to call adapter factories within eclipse framework. The rest of this sub-chapter lists advantages and disadvantages of this adapter pattern flavor.<br />
<br />
<a name="AdapterPatternThirdPartyAdapterEclipseImplementation"></a><h6>Eclipse Implementation</h6>Adapters are created by adapter factories. To achieve their independence, adapter factories are hidden behind global adapter manager. Client code never communicates directly with an adapter factory. <br />
<br />
Adapter manager exposes a <code>getAdapter(Object obj, Class type)</code> method which delegates calls to installed adapter factories. This method returns either an adapter or a <code>null</code>.<br />
<br />
Use adapter manager to create an adapter:<br />
<pre class="brush:java">(Plus) Platform.getAdapterManager().getAdapter(minus, Plus.class);
</pre><br />
Adapter factories are registered into adapter manager either programmatically or in plugin.xml. As our plugin does not need to do this, we will omit that information. Read either <a href="http://www.eclipse.org/articles/article.php?file=Article-Adapters/index.html">eclipse corner article</a> or <a href="http://www.eclipse.org/resources/resource.php?id=407">Eclipse Adapters</a> tutorial for further information.<br />
<br />
Note: the adapter factory is not allowed to call the <code>getAdapter(Class type)</code> method of converted object. It would cause an infinite loop, because the <code>getAdapter</code> method of an adaptable object may call adapter manager.<br />
<br />
<a name="AdapterPatternThirdPartyAdapterWhenToUseIt"></a><h6>When To Use It</h6>Adapter factories lead to high decoupling and extensibility. The decoupling of the original object and required type is absolute. There is no dependency between them. Instead, an independent adapter factory depends on both.<br />
<br />
Asking an adapter manager for adapter makes your feature extensible. Anyone can contribute adapters to integrate his objects with your plugin.<br />
<br />
Use it to: <br />
<ul><li>make your plugin extensible by third party plugin writers,</li>
<li>integrate two plugins.</li>
</ul><br />
<a name="AdapterPatternThirdPartyAdapterLimitations"></a><h6>Limitations</h6>The decoupling comes with a higher complexity. If something goes wrong, it may be difficult to find out where the faulty adapter came from. Similarly, finding out which adapters are available is more time consuming than in the 'adaptable objects' version of this pattern.<br />
<br />
If you expect others to extend your plugin, document which adapters are expected. Document also adapter factories you are adding into the system. Hunting down all that information in xml files can be very time consuming. <br />
<br />
<a name="AdapterPatternAdaptanObject"></a><h5>Adapt an Object</h5>The most correct way of adapting an object to the desired interface makes no assumptions about the adapter pattern version used by the object. <br />
<br />
Follow three steps to adapt the object to the needed interface or class: <br />
<ul><li>If the object implements or extends the desired interface, use the object.</li>
<li>If the object can adapt itself, use the adapter provided by the object.</li>
<li>Use the global adapter manager to adapt the object.</li>
</ul><br />
Adapt an object to a type, full implementation:<br />
<pre class="brush:java">public static Object getAdapter(Object obj, Class type) {
// if the object implements or extends the desired interface, use it
if (type.isInstance(obj))
return obj;
// if the object is able to adapt itself, let it do it
if (obj instanceof IAdaptable) {
IAdaptable adaptable = (IAdaptable) obj;
Object adapter = adaptable.getAdapter(type);
if (adapter != null)
return adapter;
}
// delegate to the global adapter manager
return Platform.getAdapterManager().getAdapter(obj, type);
}
</pre><br />
<a name="AbstractSyntaxTreevsJavaModel"></a><h4>Abstract Syntax Tree vs Java Model</h4>Eclipse framework uses two different hierarchies to represent java source code. First is called java model and second is called abstract syntax tree. These hierarchies are mostly independent and have different purposes and usage.<br />
<br />
Java model hierarchy is lightweight, fault tolerant, fast to re-create and its abilities are limited. Abstract syntax tree provides full control over java source code, but is much slower to recreate. For these reasons, Eclipse framework uses java model hierarchy wherever possible. Abstract syntax tree is used only if necessary.<br />
<br />
First sub-chapter shows how to enable these hierarchies in the plugin. Second sub-chapter contains overview of java model and last contains overview of abstract syntax tree. <br />
<br />
<a name="AbstractSyntaxTreevsJavaModelPluginDependency"></a><h5>Plugin Dependency</h5>Both hierarchies belong to the <code>org.eclipse.jdt.core</code> plugin. We have to add jdt core plugin to plugin dependencies if we want to use them: <br />
<ul><li>Open plugin.xml and go to the 'Dependencies' tab.</li>
<li>Click Add button in the Required Plug-ins section.</li>
<li>Select the <code>org.eclipse.jdt.core</code> plugin.</li>
</ul><br />
Eclipse will automatically add plugin dependency into <code>Require-Bundle</code> section of <code>MANIFEST.MF</code> file.<br />
<br />
<a name="AbstractSyntaxTreevsJavaModelJavaModel"></a><h5>Java Model</h5>Java model is a set of interfaces that represent java methods, classes, interfaces and other elements. It is lightweight, fault tolerant and fast to re-create. <br />
<br />
Java model hierarchy provides basic information about java code structure. It is also able to make simple changes on it. For example, it is possible to use it to rename or add new methods, classes or interfaces. <br />
<br />
The main disadvantage of the java model hierarchy is that it does not contain full information about underlying source code. For example, it does not contain methods bodies. As a result, it is not possible to perform more complicated source code changes.<br />
<br />
The root of the java model hierarchy is <code>IJavaElement</code> interface. All interfaces that extend it belong to this hierarchy. <br />
<br />
<a name="AbstractSyntaxTreevsJavaModelAbstractSyntaxTree"></a><h5>Abstract Syntax Tree</h5>Abstract syntax tree is a set of classes that represent java methods, classes, interfaces and other elements. It provides full information about java code structure and is able to make any changes on it.<br />
<br />
The main disadvantage of the abstract syntax tree hierarchy is that it is slower to re-create than java model. It is also less fault tolerant.<br />
<br />
The root of the abstract syntax hierarchy is <code>ASTNode</code> class. All classes that extend it belong to this hierarchy. <br />
<br />
<a name="UserInterface"></a><h4>User Interface</h4>This chapter contains quick overview of eclipse user interface. We will explain only absolute basics of those parts of the UI that are going to be needed later in this post.<br />
<br />
First sub-chapter is about the most prominent part of Eclipse UI: views and editors. Second sub-chapter deals with the selection service. <br />
<br />
<a name="UserInterfaceParts,ViewsandEditors"></a><h5>Parts, Views and Editors</h5>Two most important visual components of Eclipse user interface are views and editors. Both editors and views can show any content and present it in any form. Both can be editable or read-only, but only editor is able to keep the content in an unsaved dirty state.<br />
<br />
<a name="UserInterfaceParts,ViewsandEditorsCommonProperties"></a><h6>Common Properties</h6>Both editors and views are called parts. Eclipse documentation uses the word part as a shortcut for 'an editor or a view'.<br />
<br />
Each part must have unique id. Technically, the id is an arbitrary string. However, the convention followed by all official plugins is to prefix the id with the plugin name. As the plugin name is usually equal to plugin root package name, the convention guarantees id uniqueness.<br />
<br />
Each part must implement <code>IWorkbenchPart</code> interface. The interface was extended twice, leading to <code>IWorkbenchPart2</code> and <code>IWorkbenchPart3</code> interfaces. You may either implement them directly, or extend the default implementation called <code>WorkbenchPart</code>.<br />
<br />
All parts are <a href="#AdapterPattern">adaptable</a>. The interface <code>IWorkbenchPart</code> extends the <code>IAdaptable</code> interface. The default implementation delegates the <code>getAdapter</code> method to the global adapter manager.<br />
<br />
<a name="UserInterfaceParts,ViewsandEditorsEditors"></a><h6>Editors</h6>An editor is typically used to edit or browse a document or input object. Changes made in editor are not stored immediately. An editor with changed content is in dirty state until the save operation is invoked. If the editor is closed without saving, all unsaved changes are lost. Its default implementation is called <code>EditorPart</code>.<br />
<br />
All editors appear in the same region of the page and can not be minimized. There can be several instances of the same type of editor.<br />
<br />
Editors toolbar is shown together with the global toolbar and their menu appears to be a part of the main menu. <br />
<br />
A document or input object edited by the editor is identified using an instance of <code>IEditorInput</code> interface. <br />
<br />
<a name="UserInterfaceParts,ViewsandEditorsViews"></a><h6>Views</h6>A view is typically used to navigate a hierarchy of information, open an editor, or display additional information for the thing opened in an active editor. Modifications made in a view are saved immediately. Its default implementation is called <code>ViewPart</code>.<br />
<br />
Views can be moved to any part of the page and can be minimized. There is generally only one instance of a given view per workbench page. <br />
<br />
Each view has its own local toolbar and menu. They are also allowed to contribute buttons to the global toolbar and menu items to the main menu. <br />
<br />
<a name="UserInterfaceSelection"></a><h5>Selection</h5>Eclipse selection system is fairly standard. Each view and editor generates its own selection. The framework has also a global selection. It usually contains selection of the active part.<br />
<br />
If the plugin needs to know selected objects no matter where they come from, it has to use the global selection. If the plugin wants to integrate itself only with few views and editors, it may listen to their selections only.<br />
<br />
This chapter shows how to interpret the selection once you get it. How to get the current selection is described in an <a href="http://www.eclipse.org/articles/Article-WorkbenchSelections/article.html">Eclipse corner article</a>.<br />
<br />
<a name="UserInterfaceSelectionSelectedObjects"></a><h6>Selected Objects</h6>Current selection always implements the <code>ISelection</code> interface. That interface is extremely minimalistic. Its only method is able to tell whether the selection is empty of not. <br />
<br />
To get more detailed information, you have to cast it to one of its subtypes. Two sub-types are interesting for our plugin:<br />
<ul><li><code>ITextSelection</code> - information about selected text. If no text is selected, contains the cursor position.</li>
<li><code>IStructuredSelection</code> - contains a list of selected objects.</li>
</ul><br />
Structured selection can contain any types of objects. The best way to convert them to required classes or interfaces is to use the <a href="#AdapterPattern">adapter pattern</a>. Simple <code>instanceof</code> and cast may not be enough. Objects flying in eclipse are often adaptable to many types, but they implement or extend only some of them.<br />
<br />
<a name="EclipseMenus"></a><h4>Eclipse Menus</h4>Eclipse menu system is surprisingly rich and complicated. There are at least five menu types. The same or similarly looking menu can be invoked in multiple ways. To make it even more interesting, Eclipse has two different frameworks able to contribute new items into all those menus.<br />
<br />
Each eclipse menu framework has its own chapter and those two chapters follow this one. This chapter contains only an overview of various menu types, an overview of the menu where our plugin adds its items and overview of those two menu frameworks.<br />
<br />
<a name="EclipseMenusMenusTypes"></a><h5>Menus Types</h5>Eclipse has five menu types:<br />
<ul><li>Context menu - menu invoked by a mouse right click. Sometimes called also popup menu.</li>
<li>Main menu - menu always visible on the top of GUI. </li>
<li>Main toolbar - always visible toolbar available under the main menu. </li>
<li>View menu - menu available in a view. Click on a white down arrow in a view to invoke it. </li>
<li>View toolbar - a little toolbar available in most views. </li>
</ul><br />
Unlike views, editors do not have their own menu or toolbar. They always contribute to the main menu or to the main toolbar.<br />
<br />
<a name="EclipseMenusSourceMenu"></a><h5>Source Menu</h5>Source menu contains items like 'Organize Imports' or 'Generate Delegate methods...' and is a logical place for our two features. It was contributed by java development tools plugin (JDT). <br />
<br />
This menu is located either in the main menu or in the context menu. Both locations show it only under some circumstances. <br />
<br />
Main menu contains source menu item only if the active view or editor supports it. Following list contains examples of views and editors that causes the source menu to appear. Activate any of them to see it in main menu: <br />
<ul><li>Java Editor,</li>
<li>Type Hierarchy View,</li>
<li>Package Explorer View,</li>
<li>Project Explorer View.</li>
</ul><br />
Context menu contains source menu item if at least one selected item represents java file, class, package method or other java item. Of course, that includes text selection inside java source editor. Activate context menu with source sub-menu:<br />
<ul><li>open java source file in an editor and right click inside,</li>
<li>select java file or class in package explorer and right click.</li>
</ul><br />
The source menu can be invoked also with shortcut 'Alt+Shift+S'. <br />
<br />
<a name="EclipseMenusMenuFrameworks"></a><h5>Menu Frameworks</h5>Eclipse has two different frameworks able to contribute new items into menus:<br />
<ul><li>the actions framework,</li>
<li>the command framework.</li>
</ul><br />
The actions framework is older and deprecated. The command framework is newer, flexible and little bit more complicated. It is superior and should be used for any new functionality. <br />
<br />
The command framework is compatible with most, but not all menus. Some menus have not been rewritten yet, so you have to use the actions framework to contribute to them. For example, the source sub-menu in the main menu is compatible only with the actions framework. <br />
<br />
Some menus are incompatible with both frameworks. The source menu invoked by the shortcut CTRLS + ALT + S is such menu. It is impossible to contribute to it.<br />
<br />
Note: Impossible to contribute to menus are rare. <br />
<br />
<a name="TheCommandFramework"></a><h4>The Command Framework</h4>The command framework is able to add items into most eclipse menus. It was designed as mini model view controller and separates UI from actions to be performed. <br />
<br />
This chapter contains command framework basics. We will explain command framework components and use them to add new items into the source menu. Then, we will show how to make menu items enabled and disabled. New menu items are going to be enabled only if they would be useful on the current selection.<br />
<br />
<a name="TheCommandFrameworkOverview"></a><h5>Overview</h5>New addition to the menu is called a menu contribution. Each menu contribution needs to know where to draw new items, which items to draw and what should happen when someone clicks them.<br />
<br />
Most <a href="http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fworkbench_cmd_menus.htm">Eclipse menus</a> have unique id assigned. If you want to contribute to the menu, you have to find that id, compose so-called location URI from it and assign it to the menu contribution.<br />
<br />
Among other things, each menu contribution can add separators and <a href="http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fworkbench_cmd_menus.htm">commands</a> into the menu. Separators are those grey lines between different menu parts. Commands placed inside a menu represent clickable menu items. Each has a label, may have an icon and can be either enabled or disabled.<br />
<br />
However, the command is not able to perform an action. It is only an abstract thing meant to separate GUI from the real action. Real work is done inside command handlers. <br />
<br />
<a name="TheCommandFrameworkMenuId"></a><h5>Menu Id</h5>Menu id is an arbitrary string assigned to the menu or sub-menu. Each menu may have multiple ids. Finding menu id of an Eclipse menu can get very frustrating and difficult. <br />
<br />
If you are lucky, the id will be revealed with the <a href="http://www.vogella.de/articles/EclipseCodeAccess/ar01s03.html">Plugin Spy</a> available in Eclipse RCP. It is able to show information about menu items, UI elements and running Eclipse plugins. If you are unlucky, you will have to find the id on your own.<br />
<br />
<a name="TheCommandFrameworkMenuIdTopLevel"></a><h6>Top Level</h6>Things are easy if you wish to add an item to the menu or toolbar top level. <br />
<br />
List of general application menus:<br />
<ul><li>The main Eclipse menu uses <code>org.eclipse.ui.main.menu</code>. The id refers only to the menu top level, e.g. the item will be placed together with 'File', 'Edit' and 'Help'.</li>
<li>The Eclipse toolbar uses <code>org.eclipse.ui.main.toolbar</code>.</li>
<li>The context menu uses <code>org.eclipse.ui.popup.any</code> id.</li>
</ul><br />
If you wish to contribute to menu or toolbar inside some view, use the view id. Luckily, the Plugin Spy can help to find the view id. Open the view in an RCP Eclipse and press 'Alt + Shift + F1'. Plugin Spy will open a popup with view information:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhufqAhO70wTimGPRkuePW4vUw95vdiKBuZfnSvM6razjrhU7vHPoGFwqXrGBy9MWUjXAKTzFHNor8cB__cOwj4jjqtN8sX2dXvnoFtqHWxVyGJhq4ylWZ441Pah3cTsZAXANWFB4EY-PyI/s1600/PluginSpy+-+0.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="320" width="191" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhufqAhO70wTimGPRkuePW4vUw95vdiKBuZfnSvM6razjrhU7vHPoGFwqXrGBy9MWUjXAKTzFHNor8cB__cOwj4jjqtN8sX2dXvnoFtqHWxVyGJhq4ylWZ441Pah3cTsZAXANWFB4EY-PyI/s320/PluginSpy+-+0.png" /></a></div><br />
<a name="TheCommandFrameworkMenuIdSubmenuWithPluginSpy"></a><h6>Sub-menu With Plugin Spy</h6>Finding an id of a sub-menu can be more complicated because the Plugin Spy may not be able to find it. <br />
<br />
Use 'Alt + Shift + F2' shortcut to start the Plugin Spy and open menu you want to contribute to. Click on any menu item and Plugin Spy will show various information about it. <br />
<br />
Use Plugin Spy to get the source menu location URI:<br />
<ul><li>open any java file in the development eclipse,</li>
<li>press 'Alt + Shift + F2',</li>
<li>right click inside the java file</li>
<li>hover over the 'Source' menu item,</li>
<li>click on the 'Clean Up...' item.</li>
</ul><br />
Eclipse will show following pop-up:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiApkYcUlUWtMdarUjJOD18vZiacHiyhxlTUHin3_IMz7xtxb_ZAiPGxpC7Mhi6NBNDcfejkFP7-9UVGsg_BqOCyTSHSKpbcbfUk_hIHdhhdGRUK4jT4P2LG-j9CMuac_6Kqxp9FM9aLtLq/s1600/PluginSpy+-+1.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="272" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiApkYcUlUWtMdarUjJOD18vZiacHiyhxlTUHin3_IMz7xtxb_ZAiPGxpC7Mhi6NBNDcfejkFP7-9UVGsg_BqOCyTSHSKpbcbfUk_hIHdhhdGRUK4jT4P2LG-j9CMuac_6Kqxp9FM9aLtLq/s320/PluginSpy+-+1.png" /></a></div><br />
If the popup contains location URI, then we are almost there. Copy and paste it somewhere. Warning: Plugin Spy popup adds new line before and after copied text, so it may look like nothing was copied.<br />
<br />
The part between : and ? is the menu id. For example, if the plugin spy shows <code><br />
menu:org.eclipse.jdt.ui.source.menu?after=CleanUp</code>, the menu id is<br />
<pre class="brush:java">org.eclipse.jdt.ui.source.menu
</pre><br />
<a name="TheCommandFrameworkMenuIdTheOtherWay"></a><h6>The Other Way</h6>If the Plugin Spy does not show the location URI, it means that menu item or the menu itself was not rewritten into command framework yet. This is how Plugin Spy popup of an old menu item looks like:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiweXwXP4LB-UO0UyOiLW6-WkFDv9iaO7RpT-KHwXrp866vOcgbShvqyuJlfwaBUcDne3Q3sijsx7fv_H-cvxIO-LYZci_ACDpfguSmEpecPPoimoD4kPLCl__vQsOWyzz5u6SGoWeeQyK6/s1600/PluginSpy+-+2.png" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="239" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiweXwXP4LB-UO0UyOiLW6-WkFDv9iaO7RpT-KHwXrp866vOcgbShvqyuJlfwaBUcDne3Q3sijsx7fv_H-cvxIO-LYZci_ACDpfguSmEpecPPoimoD4kPLCl__vQsOWyzz5u6SGoWeeQyK6/s320/PluginSpy+-+2.png" /></a></div><br />
Your best bet is to google around and look into plugin.xml of plugins that contribute to that menu. Chances are, that someone had the same problem and managed to solve it. <br />
<br />
You can start with an old but still good <a href="http://www.jdg2e.com/ch21.actions.table/doc/index.html">list of context menus</a> available on jdg2e. If the menu existed already in Eclipse 3.0, it is listed there along with its menu id. The source menu is not available in that table. <br />
<br />
However, the id of its parent menu, e.g. the menu available if you right click in any java file, is <code>#CompilationUnitEditorContext</code>. This would be an acceptable second prize.<br />
<br />
<a name="TheCommandFrameworkMenuIdPlanB"></a><h6>Plan B</h6>While the contribution to top level menus with the command framework is easy, unknown menu ids make contributions to various sub-menus difficult or even impossible. If you have no luck finding it, you have to use the old <a href="#TheActionsFramework">actions framework</a>.<br />
<br />
<a name="TheCommandFrameworkLocationURI"></a><h5>Location URI</h5>Location URI string has three parts: scheme, menu id and placement modifier:<br />
<pre class="brush:xml"><scheme>:<menu-id>[?<placement-modifier>]
</pre><br />
Eclipse has three schemes:<br />
<ul><li><code>menu</code> - either main application menu or a view menu,</li>
<li><code>toolbar</code> - either main application toolbar or a view toolbar,</li>
<li><code>popup</code> - a context menu, e.g. any menu invoked by a right click.</li>
</ul><br />
The placement modifier is optional. It has the form <code><placement>=<id></code>. The placement part is either <code>after</code> or <code>before</code>. The id is either separator name, menu ID, or item ID. Be careful, if the menu does not contain specified id, it will ignore your contribution.<br />
<br />
By convention, each menu should declare a special item with id 'additions'. This is where menu authors through that you should place your items. You do not have to respect their wish and some menus do not have such item. You can place your items wherever you like, even if the special additions item is in the menu.<br />
<br />
Few examples:<br />
<ul><li><code>menu:org.eclipse.ui.main.menu?after=file</code> - add item to the main menu, between File and Edit sub-menus.</li>
<li><code>toolbar:org.eclipse.ui.main.toolbar</code> - add item to the main toolbar.</li>
<li><code>menu:org.eclipse.ui.navigator.ProjectExplorer</code> - add item to the menu available in the project explorer view.</li>
<li><code>toolbar:org.eclipse.ui.navigator.ProjectExplorer</code> - add item to the toolbar available in the project explorer view.</li>
<li><code>popup:org.eclipse.ui.popup.any</code> - add item to the context menu.</li>
</ul><br />
Place the item into the source menu without being picky about the exact location:<br />
<pre class="brush:xml">popup:org.eclipse.jdt.ui.source.menu
</pre><br />
Note: We omitted the optional exact location in our location URI, because it can be tricky. If you wish to specify also the exact location, you have to be sure that selected place exists every time the menu is invoked. Unfortunately, just because the menu item looks the same, it does not mean that it really is the same menu item. <br />
<br />
For example, if we would add <code>?after=CleanUp</code> in the end of the previous location, our menu item would be placed right after the 'Clean Up...' item in the sources sub-menu invoked by right click in java editor. However, it would be invisible in the source sub-menu invoked by right click in package explorer view. <br />
<br />
<a name="TheCommandFrameworkCommand"></a><h5>Command</h5>A command is an abstract thing that represents an action and has an unique id. It can be placed into the menu and you can assign a shortcut to it.<br />
<br />
Commands are configured in plugin.xml file. Open it and go to extensions tab. Use the left part named 'All Extensions':<br />
<ul><li>Click 'Add' and choose <code>org.eclipse.ui.commands</code> extension point.</li>
<li>Right click on the new extension point and choose 'New' -> 'command'.</li>
<li>Fill in the id. The convention is to use plugin name as prefix of the command id.</li>
<li>Fill in the name.</li>
</ul><br />
We created two commands. One named 'Generate Custom toString' and another is named 'Check Non-Null Parameter'. Their ids are:<br />
<ul><li><code>org.meri.eclipse.defensiveapitools.generatetostring</code></li>
<li><code>org.meri.eclipse.defensiveapitools.checknonnullparameter</code></li>
</ul><br />
Eclipse automatically generates corresponding xml and places it to the plugin.xml file. Go to the plugin.xml tab to see it:</div><pre class="brush:xml"><extension
point="org.eclipse.ui.commands">
<command
id="org.meri.eclipse.defensiveapitools.generatetostring"
name="Generate Custom toString">
</command>
<command
id="org.meri.eclipse.defensiveapitools.checknonnullparameter"
name="Check Non-Null Parameter">
</command>
</extension>
</pre><div style="text-align: justify;"><br />
<a name="TheCommandFrameworkMenuContribution"></a><h5>Menu Contribution</h5>Menu contribution is an addition to menu. It knows where to draw itself and contains list of items that should be drawn. All items placed into the menu contribution are shown together, in the order defined by their position in that list.<br />
<br />
We will configure a new menu contribution and add two menu items and two separators in it. Adding other element types into it is analogical.<br />
<br />
<a name="TheCommandFrameworkMenuContributionMenuContribution"></a><h6>Menu Contribution</h6>Menu items are configured in plugin.xml file. Open it and go to extensions tab. Use the left part named 'All Extensions':<br />
<ul><li>Click 'Add' and choose <code>org.eclipse.ui.menus</code> extension point.</li>
<li>Right click on the new extension point and choose 'New' -> 'menuContribution'.</li>
<li>Fill in the <a href="#TheCommandFrameworkLocationURI">locationURI</a>. </li>
<li>Set allPopups to <code>true</code>.</li>
</ul><br />
Recall, that the source context menu location uri is: <br />
<ul><li><code>popup:org.eclipse.jdt.ui.source.menu</code></li>
</ul><br />
<a name="TheCommandFrameworkMenuContributionCommand"></a><h6>Command</h6>Add commands to the menu contribution: <br />
<ul><li>Right click on the menu contribution and choose 'New' -> 'command'.</li>
<li>Click Browse and find previously created command.</li>
<li>Fill in label.</li>
</ul><br />
Eclipse automatically generates corresponding xml and places it to the plugin.xml file. Go to the plugin.xml tab to see it:</div><pre class="brush:xml"><extension
point="org.eclipse.ui.menus">
<menuContribution
allPopups="true"
locationURI="popup:org.eclipse.jdt.ui.source.menu">
<command
commandId="org.meri.eclipse.defensiveapitools.generatetostring"
label="Generate Custom toString"
style="push">
</command>
<command
commandId="org.meri.eclipse.defensiveapitools.checknonnullparameter"
label="Check Non-Null Parameter"
style="push">
</command>
</menuContribution>
</extension>
</pre><div style="text-align: justify;"><br />
Test the menu. Run or <a href="#SimplePluginProjectDebugThePlugin">debug</a> the plugin and open any java file in the test Eclipse. Right click and choose the Source sub-menu. The menu contains two new items. As our commands have no handlers assigned, both items are disabled.<br />
<br />
<a name="TheCommandFrameworkMenuContributionSeparator"></a><h6>Separator</h6>Add separator to the menu contribution: <br />
<ul><li>Right click on the menu contribution and choose 'New' -> 'separator'.</li>
<li>Set visible to <code>true</code>.</li>
<li>Fill in name.</li>
</ul><br />
The order of elements in the menu contribution defines the order of corresponding items in the menu. Drag and drop new separator to the place where you would like to see it.<br />
<br />
Eclipse automatically generates corresponding xml and places it to the plugin.xml file. Go to the plugin.xml tab to see it:</div><pre class="brush:xml"><menuContribution
allPopups="true"
locationURI="popup:org.eclipse.jdt.ui.source.menu">
<separator
name="org.meri.eclipse.defensiveapitools.begin"
visible="true">
</separator>
... commands ...
<separator
name="org.meri.eclipse.defensiveapitools.end"
visible="true">
</separator>
</menuContribution>
</pre><div style="text-align: justify;"><br />
Test the menu again. Run or <a href="#SimplePluginProjectDebugThePlugin">debug</a> the plugin and open any java file in the test Eclipse. Right click and choose the Source sub-menu. Our menu items are surrounded by menu separators.<br />
<br />
<a name="TheCommandFrameworkCommandHandler"></a><h5>Command Handler</h5>The command handler is the class that executes an action whenever users clicks on the menu item. Once we assign it to the command, the menu item created in previous chapter will be enabled.<br />
<br />
This section is divided into three part. First two sections show how to create and configure a dummy handler. Third section explains where the handler can get the information about currently selected items, active editor and other UI state.<br />
<br />
<a name="TheCommandFrameworkCommandHandlerSimplestCommandHandler"></a><h6>Simplest Command Handler</h6>A command handler must implement the <code>IHandler2</code> interface. The easiest way to implement it is to extend the abstract <code>AbstractHandler</code> class. This class provides standard implementation of all necessary methods except the <code>execute</code> method.<br />
<br />
The execute method is called whenever user invoked the command. It has one parameter which contains information about the current state of the application. The execute method must return <code>null</code>.<br />
<br />
Our first command handler is very simple:</div><pre class="brush:java">public class GenerateToStringHandler extends AbstractHandler {
@Override
public Object execute(ExecutionEvent event) throws ExecutionException {
System.out.println("GenerateToStringHandler");
// must return null
return null;
}
}
</pre><div style="text-align: justify;"><br />
<a name="TheCommandFrameworkCommandHandlerBasicConfiguration"></a><h6>Basic Configuration</h6>The only thing we have to configure is which commands should be handled by our handler. <br />
<br />
Note: It is also possible to assign multiple handlers to one command, but we will not do it. It is an advanced topic discussed in <a href="#TheCommandFrameworkCommandFrameworkAdditionalResources">additional resources</a>. <br />
<br />
As usually, command handlers are configured in plugin.xml file. Open it and go to extensions tab. Use the left part named 'All Extensions':<br />
<ul><li>Click 'Add' and choose <code>org.eclipse.ui.handlers</code> extension point.</li>
<li>Right click on the new extension point and choose 'New' -> 'handler'.</li>
<li>Browse for the command id.</li>
<li>Browse for the class implementing the command handler.</li>
</ul><br />
Eclipse automatically generates corresponding xml and places it to the plugin.xml file. Go to the plugin.xml tab to see it:</div><pre class="brush:xml"><extension
point="org.eclipse.ui.handlers">
<handler
class="org.meri.eclipse.defensiveapitools.generatetostring.GenerateToStringHandler"
commandId="org.meri.eclipse.defensiveapitools.generatetostring">
</handler>
<handler
class="org.meri.eclipse.defensiveapitools.checknonnullparameter.CheckNonNullParameterHandler"
commandId="org.meri.eclipse.defensiveapitools.checknonnullparameter">
</handler>
</extension>
</pre><div style="text-align: justify;"><br />
Run or <a href="#SimplePluginProjectDebugThePlugin">debug</a> the plugin and open any java file in the test Eclipse. Right click and choose the Source sub-menu. The 'Generate Custom toString' menu item is enabled. If you click on it, the command handler will print 'GenerateToStringHandler' into console in RCP Eclipse.<br />
<br />
<a name="TheCommandFrameworkCommandHandlerImplementation"></a><h6>Implementation</h6>As we are going to invoke our feature from two different places, our command handler will delegate all real functionality to yet another class. <br />
<br />
That another class will need some information about the context surrounding the command. Namely, 'Generate Custom toString' needs to know which class was selected and 'Check Non-Null Parameters' needs to know which method was selected.<br />
<br />
Note: Eclipse generates mini-selection every time you place the cursor somewhere. You do not have to highlight the text inside editor.<br />
<br />
The <code>execute</code> method obtains an instance of <code>ExecutionEvent</code> as a parameter. The execution event has reference to application context which contains various information about the state of the eclipse. <br />
<br />
Use the <code>HandlerUtil</code> to get that information out of the execution event. Handler util is a static class and has a lot of <code>getWHATEVER</code> methods. We will need four of them: <br />
<ul><li><code>getCurrentSelection</code> - returns current selection,</li>
<li><code>getActivePartId</code> - returns id of active view or editor,</li>
<li><code>getActiveEditorInput</code> - returns object edited by the opened editor,</li>
<li><code>getActiveShell</code> - returned object will be needed for our plugin dialog.</li>
</ul><br />
<pre class="brush:java">private static final String JAVA_EDITOR_ID =
"org.eclipse.jdt.ui.CompilationUnitEditor";
public Object execute(ExecutionEvent event) throws ExecutionException {
//this object is needed to render wizards, messages and so on
Shell activeShell = HandlerUtil.getActiveShell(event);
//get selected items or text
ISelection currentSelection = HandlerUtil.getCurrentSelection(event);
//identify active GUI part
String activePartId = HandlerUtil.getActivePartId(event);
if (JAVA_EDITOR_ID.equals(activePartId)) {
//get edited file
IEditorInput input = HandlerUtil.getActiveEditorInput(event);
//currentSelection contains text selection inside input file
//... locate class selected in that file ...
} else {
//currentSelection contains all selected classes
//... collect all selected classes ...
}
return null;
}
</pre><br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibility"></a><h5>Enabling, Disabling and Visibility</h5>The menu item does not have to be visible and enabled all the time. Most commands are not universally usable. <br />
<br />
Both our features are going to be visible all the time. However, we will have them enabled only under some conditions. 'Check Non-Null Parameters' will be enabled:<br />
<ul><li>inside java editor,</li>
<li>if the current selection contains only modifiable java methods.</li>
</ul><br />
'Generate Custom toString' will be enabled:<br />
<ul><li>inside java editor,</li>
<li>if the current selection contains only modifiable java classes or source files.</li>
</ul><br />
First, shortest section shows where to configure visibility and enablement. Next one describes xml language used to define visibility and enable conditions. Third section uses that language to test active editor and fourth tests selected objects. Final two sections put everything together and show conditions needed for our plugin.<br />
<br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityVisiblevsHidden"></a><h6>Visible vs Hidden</h6>If you want to make the menu item invisible, use the <code>visibleWhen</code> tag on a command reference inside a menu contribution. If you want to make the item disabled, use the <code>enabledWhen</code> tag on the command handler. Both tags works exactly the same way.<br />
<br />
Open plugin.xml in an editor and go to the extensions tab. Right click either on the reference inside a menu contribution or command handler and choose either <code>visibleWhen</code> or <code>enabledWhen</code>.<br />
<br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityExpressionLanguage"></a><h6>Expression Language</h6>If you right click on <code>visibleWhen</code> or <code>enabledWhen</code> tag, Eclipse will show a list of possible sub-tags. All those tags are part of xml boolean expression language and we will use them to define a condition. If the condition is satisfied, the menu item will be visible or enabled. If the condition is not satisfied, the menu item is either disabled or invisible. <br />
<br />
One warning: not each tag shown on the list is directly usable. In this case, Eclipse simply shows all tags that defines a condition. <br />
<br />
Usage of all listed tags is described on <a href="http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fextension-points%2Forg_eclipse_ui_handlers.html">Eclipse help page</a>. We will explain only five tags needed for our conditions:<br />
<ul><li><code>or</code> - logical or,</li>
<li><code>with</code> - specify the object under condition,</li>
<li><code>iterate</code> - iterator over a collection,</li>
<li><code>equals</code> - compares object under condition with a value,</li>
<li><code>adapt</code> - adapts object under condition to the specified one.</li>
</ul><br />
<code>Or</code><br />
The element <code>or</code> does logical or. Its child tags must represent conditions. If at least one of them returns true, the result is true. <br />
<br />
<code>With</code><br />
The element <code>with</code> specifies the object under condition. This tag can access various variables that describe Eclipse state. Each variable has a name. All childs of the <code>with</code> tag will test their condition against the value of the specified variable. <br />
<br />
This tag has one mandatory property <code>variable</code>. Use it to specify the variable name. We will use two variables:<br />
<ul><li>activeMenuSelection - collection of all objects selected by the user,</li>
<li>activePartId - the id of currently active GUI part (view, editor, preference page ...).</li>
</ul><br />
As any plugin can add its own variable, the complete list of all variables is impossible to find. The list of default variables is available on <a href="http://wiki.eclipse.org/Command_Core_Expressions#Variables_and_the_Command_Framework">Eclipse wiki page</a>.<br />
<br />
<code>Iterate</code><br />
The element <code>iterate</code> is usable only inside the <code>with</code> tag and only if this tag specified a collection as the object under the test. Its child tags must represent conditions. Iterate iterates through all objects inside the collection and runs all child conditions on those objects. <br />
<br />
The <code>iterate</code> tag has two arguments: <code>operator</code> and <code>ifempty</code>. The first argument value can be either <code>and</code> or <code>or</code>. Selected operator will be applied to evaluated conditions results. If no operator is specified, the iterator uses <code>and</code>.<br />
<br />
The <code>ifempty</code> can be either <code>true</code> or <code>false</code>. This value will be returned if the collection under test is empty. If not specified then <code>true</code> is returned when the operator equals <code>and</code> and <code>false</code> is return if the operator equals <code>or</code>. <br />
<br />
<code>Equals</code><br />
The element <code>equals</code> compares the object under test with its <code>value</code> argument. As the object under test may not be a string, the value argument is converted into the object. The exact conversion algorithm is described in <a href="http://help.eclipse.org/helios/topic/org.eclipse.platform.doc.isv/reference/extension-points/org_eclipse_ui_handlers.html#e.equals">Eclipse manual</a>.<br />
<br />
<code>Adapt</code><br />
The element <code>adapt</code> <a href="#AdapterPattern">adapts</a> the object under test to the interface or class specified in its <code>type</code> argument. It may have child tags that may process it further, but we will use it solely to check whether the object under test is adaptable to the desired interface.<br />
<br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityTestCurrentEditor"></a><h6>Test Current Editor</h6>If the menu was invoked from java editor, then id of the active part is the same as id of java editor. Therefore, we have to get java editor id and compare it to the value of <code>activePartId</code> expression variable.<br />
<br />
To get the id, open any java file in RCP eclipse and press ALT + SHIFT + F1. Plugin Spy will show popup with various information about active GUI part. According to this popup, the java editor id is <code>org.eclipse.jdt.ui.CompilationUnitEditor</code>.<br />
<br />
Combine the <code>with</code> and <code>equals</code> tags to compare the id:<br />
<pre class="brush:java"><with variable="activePartId">
<equals value="org.eclipse.jdt.ui.CompilationUnitEditor" />
</with>
</pre><br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityTestSelectedObjects"></a><h6>Test Selected Objects</h6>Current selection can contain multiple items and all of them should represent something we can work with. The <a href="#AdapterPattern">adapter pattern</a> and java model hierarchy have been designed exactly for these kind of situation.<br />
<br />
Three interfaces from java model hierarchy are relevant to our features:<br />
<ul><li><code>org.eclipse.jdt.core.IMethod</code> - represents java methods,</li>
<li><code>org.eclipse.jdt.core.IType</code> - represents java classes and interfaces,</li>
<li><code>org.eclipse.jdt.core.ICompilationUnit</code> - represents java source file.</li>
</ul><br />
Therefore, we will iterate over the <code>activeMenuSelection</code> variable and check whether each selected object is adaptable to one of needed types. If no item is selected, the condition should return <code>false</code>.<br />
<br />
A method is selected if it is possible to adapt all selected objects into the <code>IMethod</code> interface:<br />
<pre class="brush:java"><with variable="activeMenuSelection">
<iterate ifEmpty="false" operator="and">
<adapt type="org.eclipse.jdt.core.IMethod" />
</iterate>
</with>
</pre><br />
A java source file or a class is selected if it is possible to adapt all selected objects into either <code>ICompilationUnit</code> or <code>IType</code>interface:<br />
<pre class="brush:java"><with variable="activeMenuSelection">
<iterate ifEmpty="false" operator="and">
<or>
<adapt type="org.eclipse.jdt.core.IType" />
<adapt type="org.eclipse.jdt.core.ICompilationUnit" />
</or>
</iterate>
</with>
</pre><br />
Note 1: Unfortunately, the xml expression language is not expressive enough to distinguish between classes and interfaces. An instance of <code>IType</code> interface represents class if its <code>isClass</code> method returns <code>true</code>, but xml language does not support method calls.<br />
<br />
Note 2: We are cheating little bit here. It is not possible to modify compiled java methods and classes inside jar packages, but they are adaptable to java model interfaces too. Those objects are modifiable only if the method <code>getCompilationUnit</code> does not return <code>null</code>. As with the previous note, this is not possible to check from the xml expression language. Fortunately, we are contributing to the source sub-menu which is available only on modifiable java elements, so we do not have to solve this problem. <br />
<br />
Note 3: In both cases, the correct solution would be to create own <code>with</code> variable and handle both problems in java. This is possible and easy, but out of scope of this article. If you wish to know how to do it, read the post on <a href="http://www.vogella.de/articles/EclipseCommandsAdvanced/article.html">Lars Vogel</a> blog.<br />
<br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityCheckNonNullParametersEnablement"></a><h6>Check Non-Null Parameters Enablement</h6>To reiterate, the menu item 'Check Non-Null Parameters' will be enabled:<br />
<ul><li>inside java editor,</li>
<li>if the current selection contains only modifiable java methods.</li>
</ul><br />
Use <code>or</code> to combine conditions created in previous chapters. This is the final condition:<br />
<pre class="brush:java"><enabledWhen>
<or>
<with variable="activePartId">
<equals value="org.eclipse.jdt.ui.CompilationUnitEditor" />
</with>
<with variable="activeMenuSelection">
<iterate ifEmpty="false" operator="and">
<adapt type="org.eclipse.jdt.core.IMethod" />
</iterate>
</with>
</or>
</enabledWhen>
</pre><br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityGenerateCustomtoStringEnablement"></a><h6>Generate Custom toString Enablement</h6>Recall, that the menu item 'Generate Custom toString' will be enabled:<br />
<ul><li>inside java editor,</li>
<li>whenever java class is selected,</li>
<li>whenever java file is selected.</li>
</ul><br />
Use <code>or</code> to combine conditions created in previous chapters. This is the final condition:<br />
<pre class="brush:java"><enabledWhen>
<or>
<with variable="activePartId">
<equals value="org.eclipse.jdt.ui.CompilationUnitEditor" />
</with>
<with variable="activeMenuSelection">
<iterate ifEmpty="false" operator="and">
<or>
<adapt type="org.eclipse.jdt.core.IType" />
<adapt type="org.eclipse.jdt.core.ICompilationUnit" />
</or>
</iterate>
</with>
</or>
</enabledWhen>
</pre><br />
<a name="TheCommandFrameworkEnabling,DisablingandVisibilityExpressionLanguageAdditionalResources"></a><h6>Expression Language - Additional Resources</h6>The complete explanation on how those conditions work is out of scope of this post. If you wish to read more, the basic and short article is on <a href="http://blog.eclipse-tips.com/2009/01/commands-part-2-selection-and.html">eclipse-tips</a> blog and a detailed one is on <a href="http://www.vogella.de/articles/EclipseCommandsAdvanced/article.html">Lars Vogel</a> blog. <br />
<br />
The detailed post explains also how to reuse conditions and how to create new <code>with</code> variable. It is definitely worth reading, especially if you plan to do something complicated.<br />
<br />
<a name="TheCommandFrameworkCommandFrameworkAdditionalResources"></a><h5>Command Framework - Additional Resources</h5>We simplified things a little in this chapter. The full functionality and possibilities of the command framework are out of scope of this article.<br />
<br />
If you wish to know more, good tutorials are available on <a href="http://www.vogella.de/articles/EclipseCommands/article.html">Lars Vogel blog</a> or in <a href="http://www.ibm.com/developerworks/library/os-eclipse-3.3menu/">IBM library</a>.<br />
<br />
<a name="TheActionsFramework"></a><h4>The Actions Framework</h4>Theoretically speaking, the actions framework is old and has been deprecated. The newer commands framework should be used for most purposes. Unfortunately, not all menus have been rewritten yet. <br />
<br />
If the menu you are interested in has not been rewritten yet, you have to use the old actions framework to contribute to it. <br />
<br />
As most menus are already compatible with the new command framework, we will show only the part of actions framework that is able to contribute to the source sub-menu in main menu. If you wish to read more about the actions framework, the <a href="#TheActionsFrameworkAdditionalResources">last sub-chapter</a> links to more detailed articles.<br />
<br />
<a name="TheActionsFrameworkOverview"></a><h5>Overview</h5>Two most important elements in the actions framework are action and action delegate. An action is assigned to menu items and knows the menu where it should be drawn. Action delegate is a class that does all the work whenever a menu item is invoked. <br />
<br />
Each action delegate can be assigned to any number of actions. However, each action can have only one action delegate. <br />
<br />
Items and menus definitions are kept either inside an action set or inside something called a contribution. There is not much difference between them, both action set and contribution contain list of actions and sub-menus to be drawn within menu system. <br />
<br />
Eclipse framework has three types of actions contributions and each is able to specify a simple condition that must be satisfied to have a visible action:<br />
<ul><li><code>viewerContribution</code> - an action is available only inside specified view,</li>
<li><code>objectContribution</code> - an action is available only if specified object type has been selected,</li>
<li><code>editorContribution</code> - an action is available only inside editor menu or toolbar.</li>
</ul><br />
Note: it is also possible to assign a condition to an action. <a href="http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fworkbench_actionfilters.htm">Action conditions</a> are powerful, but out of scope of this article. <br />
<br />
Both contribution and action set are placed directly inside an extension point. Each menu type has its own extension point: <br />
<ul><li><code>org.eclipse.ui.actionSets</code> - main menu or main toolbar,</li>
<li><code>org.eclipse.ui.popupMenus</code> - context menu,</li>
<li><code>org.eclipse.ui.viewActions</code> - local menu or toolbar in a view,</li>
<li><code>org.eclipse.ui.editorActions</code> - local menu or toolbar in an editor (recall that the editors menu is shown inside the main menu/toolbar),</li>
<li><code>org.eclipse.ui.perspectiveExtensions</code> - we will ignore that.</li>
</ul><br />
So, we have to create action delegate class, identify the menu we want to contribute to and configure an action inside a action set.<br />
<br />
<a name="TheActionsFrameworkImplementingActionDelegate"></a><h5>Implementing Action Delegate</h5>Action delegate implementation depends on where you want to place the associated action. Menus supply the action delegate with additional information that depends on the menu type. Therefore, each menu type requires it to implement different interface. <br />
<br />
<a name="TheActionsFrameworkImplementingActionDelegateCommon"></a><h6>Common</h6>Each action delegate must implement <code>IActionDelegate</code> interface. That interface was later extended with some lifecycle methods, so if you need to initialize your action delegate or know when it is disposed, implement also <code>IActionDelegate2</code> interface.<br />
<br />
Which method is invoked when a menu item is clicked depends on implemented interfaces. An action delegate that does NOT implement <code>IActionDelegate2</code> interface must place all the work inside the <code>run</code> method. However, an action delegate that implements the <code>IActionDelegate2</code> interface must place all the work inside the <code>runWithEvent</code>. In this case, the <code>run</code> method is never called.<br />
<br />
<a name="TheActionsFrameworkImplementingActionDelegateMenuTypesandSubInterfaces"></a><h6>Menu Types and Sub-Interfaces</h6>Which sub-interfaces of <code>IActionDelegate</code> interface you should use depends on where do you want to place the action. Some parent menus send additional information about active parts and the action delegate must be able to acquire it.<br />
<br />
List of eclipse menus and expected action delegate interfaces:<br />
<ul><li>main menu or toolbar - <code>IWorkbenchWindowActionDelegate</code>,</li>
<li>view menu or toolbar - <code>IViewActionDelegate</code>,</li>
<li>view context menu - <code>IViewActionDelegate</code> or <code>IObjectActionDelegate</code>,</li>
<li>editor menu or toolbar - <code>IEditorActionDelegate</code>,</li>
<li>editor context menu - <code>IObjectActionDelegate</code>.</li>
</ul><br />
<a name="TheActionsFrameworkImplementingActionDelegateSelection"></a><h6>Selection</h6>Parent menu uses the method <code>selectionChanged</code> to inform the action delegate about current selection changes. However, selection changes are sent to the action delegate only after it was invoked for the first time. All prior selection changes are ignored. <br />
<br />
The <code>selectionChanged</code> method is called for the first time right before the first call of <code>run</code> or <code>runWithEvent</code> method.<br />
<br />
<a name="TheActionsFrameworkImplementingActionDelegateImplementation"></a><h6>Implementation</h6>We are ready to implement our action delegates. As it was with the command handler in command framework, our action delegate will only collect information about current editor and selected objects or text. The real work is delegated to yet another class.<br />
<br />
As we want to put our action to the main menu, we have to implement the <code>IWorkbenchWindowActionDelegate</code> interface. Main menu sends changes of current selection it to its action delegates, so all we have to do is to store it:<br />
<pre class="brush:java">private ISelection selection;
public void selectionChanged(IAction action, ISelection selection) {
this.selection = selection;
}
</pre><br />
Getting current editor is little bit more complicated. Main menu does not inform its items about active UI parts. However, it sends them an instance of <code>IWorkbenchWindow</code> upon initialization. Fortunately, the workbench window object is aware of almost everything that is going on in the eclipse.<br />
<br />
The explanation of the workbench window object would take too much space and it is not that important. The important is, that it provides access to all kind of information about eclipse user interface. <br />
<br />
Use workbench window to get active editor and its id: <br />
<pre class="brush:java">private IWorkbenchWindow window;
public void init(IWorkbenchWindow window) {
this.window = window;
}
private String getActivePartId() {
return window.getPartService().getActivePartReference().getId();
}
private IEditorPart getActiveEditor() {
return window.getActivePage().getActiveEditor();
}
</pre><br />
Finally, we are ready to implement the run method:<br />
</div><pre class="brush:java">public class GenerateToStringActionDelegate implements IWorkbenchWindowActionDelegate {
private static final String JAVA_EDITOR_ID =
"org.eclipse.jdt.ui.CompilationUnitEditor";
public void run(IAction action) {
//this object is needed to render wizards, messages and so on
Shell activeShell = window.getShell();
//get selected items or text
ISelection currentSelection = selection;
//identify active GUI part
String activePartId = getActivePartId();
//java editor must be handled differently than view selection
if (JAVA_EDITOR_ID.equals(activePartId)) {
//get edited file
IEditorInput input = getActiveEditor().getEditorInput();
//currentSelection now contains text selection inside input file
//... locate class selected in that file ...
} else {
//currentSelection now contains all classes inside
//... collect all selected classes ...
}
System.out.println("GenerateToStringActionDelegate");
}
}
</pre><div style="text-align: justify;"><br />
<a name="TheActionsFrameworkMenubarPathandToolbarPath"></a><h5>MenubarPath and ToolbarPath</h5>Before you configure the action, you have to find the menu or toolbar path. It identifies a menu or toolbar where your action will be shown. <br />
<br />
Each menu or sub-menu has its name and menu or toolbar path navigates those names. It starts with the top level menu name and continues with all sub-menu names. The final part is optional and specifies a location in the final menu. If it is missing, the item is placed to the end of the specified menu.<br />
<br />
For example, the menu path <code>#menuName/subMenuName/additions</code> should be read as "place the item in the end of the <code>additions</code> group which is located inside the sub-menu <code>subMenuName</code> of the <code>#menuName</code> menu".<br />
<br />
Or, the menu path <code>#menuName/subMenuName/</code> should be read as "place the item in the end of the sub-menu <code>subMenuName</code> of the <code>#menuName</code> menu".<br />
<br />
Important: Menu ids are the same as described in the <a href="#TheCommandFrameworkMenuId">command framework chapter</a>. The id of the source menu id is <code>org.eclipse.jdt.ui.source.menu</code>. As the main source menu is a top level menu, menubarPath is:<br />
<pre class="brush:java">org.eclipse.jdt.ui.source.menu/
</pre><br />
<a name="TheActionsFrameworkConfiguringanAction"></a><h5>Configuring an Action</h5>Action configuration is quite simple. Add an extension point, put either action set or contribution into it and place the action. <br />
<br />
Which one of five extension points should be used depends on the menu. We want to add an action to the main menu, so we have to use the action sets extension point. <br />
<br />
Open plugin.xml file and go to extensions tab. Use the left part named 'All Extensions'. First, configure an action set: <br />
<ul><li>Click 'Add' and choose <code>org.eclipse.ui.actionSets</code> extension point.</li>
<li>Right click on the new extension point and choose 'New' -> 'actionSet'.</li>
<li>Fill in the action set id. The convention is to use plugin name as its prefix. Important: This field has some strange limitations. We could not really figure them out, so all we can say is that:<ul><li><code>org.meri.eclipse.defensiveapitools.mainmenucontrib</code> works</li>
<li><code>org.meri.eclipse.defensiveapitools.sourcemenu</code> works,<br />
<li><code>org.meri.eclipse.defensiveapitools.mainsource</code> does not work,<br />
<li><code>org.meri.eclipse.defensiveapitools.actionset</code> does not work.<br />
</ul>In any case, if nothing shows up in the menu, try to change this id.</li>
<li>Fill in label. It can be anything.</li>
<li>Set visible to <code>true</code>.</li>
</ul><br />
Second, add actions to the action set: <br />
<ul><li>Right click on the action set and choose 'New' -> 'action'.</li>
<li>Fill in the action id. The convention is to use plugin name as id prefix.</li>
<li>Fill in label. It can be anything.</li>
<li>Fill in either menubarPath or toolbarPath.</li>
<li>Scroll down and browse for the action delegate in class field.</li>
</ul><br />
Eclipse automatically generates corresponding xml and places it to the plugin.xml file. Go to the plugin.xml tab to see it:<br />
<pre class="brush:xml"><extension
point="org.eclipse.ui.actionSets">
<actionSet
id="org.meri.eclipse.defensiveapitools.sourcemenu"
label="Defensive API Tools "
visible="true">
<action
class="org.meri.eclipse.defensiveapitools.generatetostring.GenerateToStringActionDelegate"
id="org.meri.eclipse.defensiveapitools.generatecustomtostring"
label="Generate Custom toString"
menubarPath="org.eclipse.jdt.ui.source.menu/"
style="push">
</action>
<action
class="org.meri.eclipse.defensiveapitools.checknonnullparameter.CheckNonNullParameterActionDelegate"
id="org.meri.eclipse.defensiveapitools.checknonnullparameters"
label="Check Non-Null Parameters"
menubarPath="org.eclipse.jdt.ui.source.menu/"
style="push">
</action>
</actionSet>
</extension>
</pre><br />
Run or <a href="#SimplePluginProjectDebugThePlugin">debug</a> the plugin and open any java file in the test Eclipse. Click on the Source main menu item. The menu contains two new items. Click on the Generate Custom toString' menu item, the action delegate will print 'GenerateToStringActionDelegate' into console in RCP Eclipse.<br />
<br />
<a name="TheActionsFrameworkAdditionalResources"></a><h5>Additional Resources</h5>Walkthrough of all types of actions contributions is available in an <a href="http://www.eclipse.org/articles/article.php?file=Article-action-contribution/index.html">eclipse corner article</a>. Easy to read <a href="http://wiki.eclipse.org/FAQ_How_do_I_add_actions_to_the_main_menu%3F">FAQ article</a> explains actions in the main menu. <br />
<br />
Actions framework supports also complicated conditions and filters to show actions only under some circumstances. If you have a reason to use that, the best starting point is in <a href="http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fworkbench_actionfilters.htm">eclipse documentation</a>.<br />
<br />
<a name="UsefulResources"></a><h4>Useful Resources</h4>Everything directly related to chapters in this post was already linked. We will add only a link on a very good series with various <a href="http://www.dolejsky.com/2007/12/16/tips-on-developing-eclipse-plugins-i/">eclipse development tips</a>.<br />
<br />
If you have a problem and can not find an answer, you can also ask questions on official freenode <a href="http://wiki.eclipse.org/IRC#Main_Channels">#eclipse</a> IRC channel. <br />
<br />
<a name="ToBeContinued"></a><h4>To Be Continued</h4>Although we added only some new items into the menu so far, almost everything needed to finish the plugin is already there. If you do not insist on having the generated code formatted, you should be able to finish the generate toString feature just by browsing through java model hierarchy API. <br />
<br />
Next part of this tutorial will show how to finish both features, including the formatting of the generated code. It will also explain how to create and work with abstract syntax tree and how to create a dialog to communicate with user. </div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com106tag:blogger.com,1999:blog-3355333693246614846.post-39170420126267924992012-03-01T00:59:00.000-08:002012-03-01T00:59:44.737-08:00JPA Tutorial<style type="text/css">
div.answertext {
border-width: .2em;
border-style: solid;
border-color: #C7BB9D;
padding-left:25px;
padding-right:25px;
}
div.answer {
border-style: none;
margin-bottom:25px;
}
div.question {
}
</style><div style="text-align: justify;">Java persistent API is a <a href="http://download.oracle.com/otndocs/jcp/persistence-2.0-fr-eval-oth-JSpec/">specification</a> and set of interfaces that helps with persistence of java objects into the database. It has been created after success of object relational mapping frameworks - the most popular is Hibernate. <br />
<br />
JPA defines standard basic set of features that every object relational mapping framework should have. Almost all current O/R frameworks implement JPA specification. Of course, most of them have additional features to set them apart.<br />
<br />
Saving/loading data with JPA is easier, faster and less error prone than writing SQL and mapping query results to objects. However, JPA does not free you from relational database understanding. And while you do not have to write SQL anymore, you have to learn JPQL which is very similar.<a name='more'></a><br />
<br />
First three chapters of this post contain configuration and set up. <a href="#Entity">Following</a> three chapters explain what are entities and show how to create, save and load them. <a href="#Relationships">The rest</a> of the post explains how to create and configure relationships between entities.<br />
<br />
<a href="" name="TableofContents"></a><h4>Table of Contents</h4><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><ul><li><a href="#DemoProject">Demo Project</a></li>
<li><a href="#JPAImplementation">JPA Implementation</a></li>
<ul><li><a href="#JPAImplementationDependency">Dependency</a></li>
<li><a href="#JPAImplementationPlugin">Plugin</a></li>
<li><a href="#JPAImplementationEclipse">Eclipse</a></li>
</ul><li><a href="#Configuration">Configuration</a></li>
<ul><li><a href="#ConfigurationOverview">Overview</a></li>
<li><a href="#ConfigurationExample">Example</a></li>
</ul><li><a href="#Entity">Entity</a></li>
<li><a href="#EntityManagerFactory">Entity Manager Factory</a></li>
<li><a href="#EntityManager">Entity Manager</a></li>
<ul><li><a href="#EntityManagerObtaining">Obtaining</a></li>
<li><a href="#EntityManagerLoading">Loading</a></li>
<li><a href="#EntityManagerDetachingEntities">Detaching Entities</a></li>
<li><a href="#EntityManagerMerge">Merge</a></li>
<li><a href="#EntityManagerUpdates,InsertsandDeletes">Updates, Inserts and Deletes</a></li>
<ul><li><a href="#EntityManagerUpdates,InsertsandDeletesOverview">Overview</a></li>
<li><a href="#EntityManagerUpdates,InsertsandDeletesCommitandRollback">Commit and Rollback</a></li>
<li><a href="#EntityManagerUpdates,InsertsandDeletesRuntimeErrors">Exceptions</a></li>
<li><a href="#EntityManagerUpdates,InsertsandDeletesUpdate">Update</a></li>
<li><a href="#EntityManagerUpdates,InsertsandDeletesInsert">Insert</a></li>
<li><a href="#EntityManagerUpdates,InsertsandDeletesDelete">Delete</a></li>
</ul><li><a href="#EntityManagerRefresh">Refresh</a></li>
</ul><li><a href="#Relationships">Relationships</a></li>
<ul><li><a href="#RelationshipsLazyLoading">Lazy Loading</a></li>
<li><a href="#RelationshipsBidirectionalvsUnidirectional">Bidirectional vs Unidirectional</a></li>
<ul><li><a href="#RelationshipsBidirectionalvsUnidirectionalSelfConsistentEntities">Self-Consistent Entities</a></li>
</ul><li><a href="#RelationshipsOwningSidevsInverseSide">Owning Side vs Inverse Side</a></li>
<li><a href="#RelationshipsPersistingRelationships">Persisting Relationships</a></li>
<ul><li><a href="#RelationshipsPersistingRelationshipsRelationshipChanges">Relationship Changes</a></li>
<li><a href="#RelationshipsPersistingRelationshipsEntityDependencies">Entity Dependencies</a></li>
<li><a href="#RelationshipsPersistingRelationshipsDeletedEntities">Deleted Entities</a></li>
<li><a href="#RelationshipsPersistingRelationshipsMergingNewEntities">Merging New Entities</a></li>
</ul><li><a href="#RelationshipsCascading">Cascading</a></li>
<ul><li><a href="#RelationshipsCascadingConfiguration">Configuration</a></li>
<li><a href="#RelationshipsCascadingLimitation">Limitation</a></li>
<li><a href="#RelationshipsCascadingMerge">Merge Problem</a></li>
<li><a href="#RelationshipsCascadingCaution">Caution</a></li>
</ul><li><a href="#RelationshipsConfigurationOverview">Configuration Overview</a></li>
<ul><li><a href="#RelationshipsConfigurationOverviewOneToOne">OneToOne</a></li>
<li><a href="#RelationshipsConfigurationOverviewOneToMany">OneToMany</a></li>
<li><a href="#RelationshipsConfigurationOverviewManyToOne">ManyToOne</a></li>
<li><a href="#RelationshipsConfigurationOverviewManyToMany">ManyToMany</a></li>
</ul><li><a href="#RelationshipsOneToOne">One-To-One</a></li>
<ul><li><a href="#RelationshipsOneToOneBasicConfiguration">Basic Configuration</a></li>
<li><a href="#RelationshipsOneToOneDefaultDatabaseStructure">Default Database Structure</a></li>
<li><a href="#RelationshipsOneToOneOptional">Optional</a></li>
<li><a href="#RelationshipsOneToOneLoading">Loading</a></li>
<li><a href="#RelationshipsOneToOneCascading">Cascading</a></li>
<li><a href="#RelationshipsOneToOneConsistency">Consistency</a></li>
<li><a href="#RelationshipsOneToOneCustomJoinColumn">Custom Join Column</a></li>
<li><a href="#RelationshipsOneToOneCustomJoinTable">Custom Join Table</a></li>
<li><a href="#RelationshipsOneToOneJoinTroughPrimaryKey">Join Trough Primary Key</a></li>
</ul><li><a href="#RelationshipsBidirectionalOneToManyManyToOne">Bidirectional One-To-Many / Many-To-One</a></li>
<ul><li><a href="#RelationshipsBidirectionalOneToManyManyToOneBasicConfiguration">Basic Configuration</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneDefaultDatabaseStructure">Default Database Structure</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneLoading">Loading </a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneCascading">Cascading</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneConsistency">Consistency</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneOptional">Optional</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneOrphanRemoval">Orphan Removal</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneOrdering">Ordering</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneCustomJoinColumn">Custom Columns</a></li>
<li><a href="#RelationshipsBidirectionalOneToManyManyToOneCustomJoinTable">Custom Join Table</a></li>
</ul><li><a href="#RelationshipsUnidirectionalManyToOne">Unidirectional Many-To-One</a></li>
<li><a href="#RelationshipsUnidirectionalOneToMany">Unidirectional One-To-Many</a></li>
<ul><li><a href="#RelationshipsUnidirectionalOneToManyBasicConfiguration">Basic Configuration</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyDefaultDatabaseStructure">Default Database Structure</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyLoading">Loading</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyCascading">Cascading</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyConsistency">Consistency</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyOrphanRemoval">Orphan Removal</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyCustomJoinColumn">Custom Join Column</a></li>
<li><a href="#RelationshipsUnidirectionalOneToManyCustomJoinTable">Custom Join Table</a></li>
</ul><li><a href="#RelationshipsManyToMany">Many-To-Many</a></li>
<ul><li><a href="#RelationshipsManyToManyBasicConfiguration">Basic Configuration</a></li>
<li><a href="#RelationshipsManyToManyDefaultDatabaseStructure">Default Database Structure</a></li>
<li><a href="#RelationshipsManyToManyCascading">Cascading</a></li>
<li><a href="#RelationshipsManyToManyLoading">Loading</a></li>
<li><a href="#RelationshipsManyToManyConsistency">Consistency</a></li>
<li><a href="#RelationshipsManyToManyOrdering">Ordering</a></li>
<li><a href="#RelationshipsManyToManyCustomJoinColumn">Custom Join Column</a></li>
<li><a href="#RelationshipsManyToManyCustomJoinTable">Custom Join Table</a></li>
</ul></ul><li><a href="#End">End</a></li>
</ul></div></div><a href="" name="DemoProject"></a><h4>Demo Project</h4>All examples and test cases used in this tutorial are available in a <a href="https://github.com/SomMeri/org.meri.jpa.tutorial">project on Github</a>. It is a plain Maven application, no configuration is necessary. <br />
<br />
The project uses an in-memory Derby database which is created before each test case and destroyed after each test case. It is lightweight and leaves no traces behind it.<br />
<br />
We used <a href="http://www.liquibase.org/">Liquibase</a> to maintain the database structure, mostly because it is practical for project that has to drop and re-create the database often. It keeps both database description and initial data in a 'changelog' xml file. <br />
<br />
Our changelog files are named <code>db.*.changelog.xml</code> and are always stored in the same directory as the test case that uses them.<br />
<br />
Other set-up details are not necessary to understand. If you are interested in them anyway, we described them in an independent <a href="http://meri-stuff.blogspot.com/2012/01/running-jndi-and-jpa-without-j2ee.html">post</a>. <br />
<br />
<a href="" name="JPAImplementation"></a><h4>JPA Implementation</h4>There are plenty of JPA implementations and they should all work the same way. We used <a href="http://openjpa.apache.org/">OpenJPA</a> in our demo project, mostly because it is free, open source and easy to set up.<br />
<br />
This chapter contains OpenJPA specific configuration. Everything else in article works the same way regardless of chosen JPA provider.<br />
<br />
First, define OpenJPA version property in <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/pom.xml">pom.xml</a>:<br />
<pre class="brush:xml"><properties>
<openjpa.version>2.1.1</openjpa.version>
</properties>
</pre><br />
<a href="" name="JPAImplementationDependency"></a><h6>Dependency</h6>Add OpenJPA dependency into pom.xml:<br />
<pre class="brush:xml"><dependency>
<groupId>org.apache.openjpa</groupId>
<artifactId>openjpa-all</artifactId>
<version>${openjpa.version}</version>
</dependency>
</pre><br />
<a href="" name="JPAImplementationPlugin"></a><h6>Plugin</h6>OpenJPA requires openjpa-maven-plugin maven plugin. It runs at compile time and modifies compiled java classes to work with OpenJPA. This process is called <a href="http://openjpa.apache.org/entity-enhancement.html">enhancement</a>.<br />
<br />
Openjpa-maven-plugin must know about all compiled entity classes. What entities are will be explained <a href="#Entity">later in this article</a>. For now, you can use a simple rule: if it has JPA annotation on it, openjpa-maven-plugin must know about it.<br />
<br />
Add openjpa-maven-plugin into pom.xml and specify all entity classes inside the <code>includes</code> sub-tag. The tag supports *:<br />
<pre class="brush:xml"><plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>openjpa-maven-plugin</artifactId>
<version>1.2</version>
<configuration>
<!-- beginning of directories with compiled entity classes -->
<includes>org/meri/jpa/*/entities/*/*.class</includes>
<includes>org/meri/jpa/*/entities/*.class</includes>
<!-- end of directories with compiled entity classes -->
<addDefaultConstructor>true</addDefaultConstructor>
<enforcePropertyRestrictions>true</enforcePropertyRestrictions>
</configuration>
<executions>
<execution>
<id>enhancer</id>
<phase>process-classes</phase>
<goals>
<goal>test-enhance</goal>
<goal>enhance</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>org.apache.openjpa</groupId>
<artifactId>openjpa</artifactId>
<version>${openjpa.version}</version>
</dependency>
</dependencies>
</plugin>
</pre><br />
<a href="" name="JPAImplementationEclipse"></a><h6>Eclipse</h6>If you use Eclipse with m2e plugin, you have to add also following huge thing into your pom.xml. It removes the 'Plugin execution not covered by lifecycle configuration' error:<br />
<pre class="brush:xml"><!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>
org.codehaus.mojo
</groupId>
<artifactId>
openjpa-maven-plugin
</artifactId>
<versionRange>
[1.0,)
</versionRange>
<goals>
<goal>test-enhance</goal>
<goal>enhance</goal>
</goals>
</pluginExecutionFilter>
<action>
<execute><runOnIncremental>true</runOnIncremental></execute>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</pre><br />
<a href="" name="Configuration"></a><h4>Configuration</h4>JPA configuration is kept in <code>META-INF/persistence.xml</code> file. If you are using Maven, the full path is <code>src/main/resources/META-INF/persistence.xml</code> .<br />
<br />
<a href="" name="ConfigurationOverview"></a><h6>Overview</h6>The persistence.xml file contains description of one or more persistence units. For readability sake, we omitted schema location and version from the next example. Full version is available <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/main/resources/META-INF/persistence.xml#L2">on Github</a>:</div><pre class="brush:xml"><?xml version="1.0" encoding="UTF-8" ?>
<persistence xmlns:xsi="..." xsi:schemaLocation="..."version="2.0" xmlns="...">
<persistence-unit name="Name1" transaction-type="RESOURCE_LOCAL">
...
</persistence-unit>
<persistence-unit name="NameN" transaction-type="RESOURCE_LOCAL">
...
</persistence-unit>
</persistence></pre><div style="text-align: justify;"><br />
Persistence unit roughly corresponds to a database and most applications have only one persistence unit. Each persistence unit configuration specifies: <br />
<ul><li>JPA persistence provider to be used,</li>
<li>database connection,</li>
<li>list of all entities,</li>
<li>other vendor specific configuration properties.</li>
</ul><br />
Each JPA implementation have different persistence provider. For example, OpenJPA uses <code>org.apache.openjpa.persistence.PersistenceProviderImpl</code> and Hibernate uses <code>org.hibernate.ejb.HibernatePersistence</code>. <br />
<br />
Each entity must be listed separately, the <code>*</code> is not supported.<br />
<br />
<a href="" name="ConfigurationExample"></a><h6>Example</h6>Our example persistence unit is named 'Simplest'. It uses OpenJPA provider and connects to the database via data source stored in JNDI. The persistence unit contains two entities <code>Person</code> and <code>AnotherEntity</code>. It also logs all queries applied to the database:</div><pre class="brush:xml"><persistence-unit name="Simplest" transaction-type="RESOURCE_LOCAL">
<!-- OpenJPA persistence provider -->
<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
<!-- JNDI name of the datasource -->
<jta-data-source>jdbc/jpaDemoDB</jta-data-source>
<!-- JPA entities must be registered here -->
<class>org.meri.jpa.simplest.entities.Person</class>
<class>org.meri.jpa.simplest.entities.AnotherEntity</class>
<!-- other properties -->
<properties>
<!-- Log all queries performed against the database. -->
<!-- Do not use in production, this will generate a lot of output. -->
<property name="openjpa.Log" value="SQL=TRACE"/>
</properties>
</persistence-unit>
</pre><div style="text-align: justify;"><br />
<a href="" name="Entity"></a><h4>Entity</h4>Entity is an ordinary java class with properties, getters and setters. It has to be marked as an entity with the <code>@Entity</code> annotation and must belong to some persistence unit. It must have a no arguments constructor.<br />
<br />
A simplest possible entity has no relationships to other entities and corresponds to one database table. Each its property corresponds to a database column. Each entity must have unique id marked with the <code>@Id</code> annotation. Of course, unique id may be composed of multiple columns. <br />
<br />
It is not possible to change the id once it was assigned. JPA behavior is undefined if this occurs.<br />
<br />
JPA assumes that the table name is the same as entity class name. Use the <code>@Table</code> annotation to customize the table name. JPA also assumes that the column name is the same is the property name. Use the <code>@Column</code> annotation to customize it. <br />
<br />
Following entity 'Person' corresponds to database table named 'person'. The <code>@Table</code> annotation is not necessary in this case, the names are the same. The entity has unique id 'id' which corresponds to the 'user_id' column. All remaining properties correspond to columns with the default name.<br />
<pre class="brush:java">@Entity
@Table(name = "Person")
public class Person {
@Id
@Column(name = "user_id")
private long id;
private String userName;
private String firstName;
private String lastName;
private String homePage;
private String about;
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
// all remaining getters and setters
}
</pre>Note: <code>@Table</code> annotation allows also customization of db schema and catalog.<br />
<br />
Person registration in 'Simples' persistence unit in persistence.xml file:<br />
<pre class="brush:xml"><persistence-unit name="Simplest" transaction-type="RESOURCE_LOCAL">
<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
<jta-data-source>jdbc/jpaDemoDB</jta-data-source>
<!-- JPA entities must be registered here -->
<class>org.meri.jpa.simplest.entity.Person</class>
</persistence-unit>
</pre><br />
Corresponding database structure:<br />
<pre class="brush:sql">CREATE TABLE person (
user_id INT NOT NULL,
username VARCHAR(1500) NOT NULL,
firstname VARCHAR(1500),
lastname VARCHAR(1500),
homepage VARCHAR(1500),
about VARCHAR(1500),
CONSTRAINT
PK_PERSON PRIMARY KEY (user_id),
UNIQUE (username)
);
</pre><br />
<a href="" name="EntityManagerFactory"></a><h4>Entity Manager Factory</h4>Each application should have one entity manager factory per persistence unit. Entity manager factory is thread safe. It should open while application starts and close on application shutdown. As the name suggests, entity manager factory is able to create an entity manager. <br />
<br />
Create entity manager factory corresponding to the 'Simplest' persistence unit:<br />
<pre class="brush:java">private static final String PERSISTENCE_UNIT = "Simplest";
private EntityManagerFactory createFactory() {
return Persistence.createEntityManagerFactory(PERSISTENCE_UNIT);
}
</pre><br />
<a href="" name="EntityManager"></a><h4>Entity Manager</h4>Entity manager is responsible for loading, saving and otherwise managing entities. It keeps track of all loaded entities in so-called persistence context. Neither entity manager nor entities are thread safe. <br />
<br />
This chapter shows most important entity manager features. First, we will show how to use it to load entities. Then, we will explain how to remove entities form the entity manager and how to add new entities into it. Finally, we will show how to modify the database and how to force reload of an entity. <br />
<br />
All examples used in this chapter are available in <code>LoadEntityTest</code> class <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/test/java/org/meri/jpa/simplest/LoadEntityTest.java">on Github</a>.<br />
<br />
<a href="" name="EntityManagerObtaining"></a><h5>Obtaining</h5>Obtain the entity manager from the entity manager factory:<br />
<pre class="brush:java">EntityManager em = getFactory().createEntityManager();
</pre><br />
<a href="" name="EntityManagerLoading"></a><h5>Loading</h5>Entity manager generates SQL statements to load entities, creates entity objects and composes them into object trees. All loaded entities are cached in something called persistence context. In other words, loaded entities are attached to entity manager. <br />
<br />
One benefit of caching in the persistence context is that entities with the same ids correspond to equal objects. <br />
<br />
Next example uses the <code>find</code> method to load entities by their ids. Loaded entities are equal:<br />
<pre class="brush:java">private static final BigDecimal SIMON_SLASH_ID = BigDecimal.valueOf(1);
@Test
public void entityEquality_same_EntityManager() {
EntityManager em = factory.createEntityManager();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
Person person2 = em.find(Person.class, SIMON_SLASH_ID);
em.close();
//entities are equal
assertEquals(person1, person2);
//changes in one entity are visible in second entity
person1.setFirstName("nobody");
assertEquals("nobody", person2.getFirstName());
}
</pre><br />
Different entity managers are completely independent of each other. If you use two different entity managers to load entities, they will result in two different independent objects.<br />
<br />
Load two entities with two entity managers and compare them. They are not the same:<br />
<pre class="brush:java">@Test
public void entityEquality_different_EntityManager() {
EntityManager em1 = factory.createEntityManager();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
em1.close();
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
em2.close();
//entities are different
assertNotSame(person1, person2);
//entities are independent
person1.setFirstName("nobody");
assertNotSame("nobody", person2.getFirstName());
}
</pre><br />
<b>Beware:</b> Be careful about closing entity managers. It automatically detaches all entities. Detached entity looses access to all JPA features. For example, <a href="#RelationshipsLazyLoading">lazy loading</a> on detached entities does not work. <br />
<br />
<a href="" name="EntityManagerDetachingEntities"></a><h5>Detaching Entities</h5>Closing entity manager is not the only way how to detach all entities. Use the <code>clear</code> method on the entity manager to detach all entities without closing it.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">/**
* Test the clear method of entity manager. Load an entity,
* clear the entity manager and load the same entity again.
* Loaded entities are different.
*/
@Test
public void entityEquality_cleared_EntityManager() {
EntityManager em = factory.createEntityManager();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
em.clear();
Person person2 = em.find(Person.class, SIMON_SLASH_ID);
em.close();
assertNotSame(person1, person2);
}
</pre><br />
</div></div>If you wish to detach only one entity, use the method <code>detach</code>:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">/**
* Test the detach method of entity manager. Load an entity,
* detach it and load the same entity again.
* Loaded entities are different.
*/
@Test
public void entityEquality_detach() {
EntityManager em = factory.createEntityManager();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
em.detach(person1);
Person person2 = em.find(Person.class, SIMON_SLASH_ID);
em.close();
assertNotSame(person1, person2);
}
</pre><br />
</div></div><a href="" name="EntityManagerMerge"></a><h5>Merge</h5>It is not possible to re-attach detached entity. However, it is possible to merge its data back to the entity manager. <br />
<br />
Merge will not attach the entity, it only copies all its data into entity manager. Changed data are not available in different entity manager:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java;">/**
* The merge method should copy all entity data into
* entity manager. Entity itself in not attached nor
* persisted.
*/
@Test
public void entityEquality_mergeInto_EntityManager() {
//load and detach entity
EntityManager em1 = factory.createEntityManager();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
em1.close();
//change its property
person1.setFirstName("New Name");
//merge it into some entity manager, data are copied
EntityManager em2 = factory.createEntityManager();
em2.merge(person1);
//this change will be ignored
person1.setFirstName("Ignored Change");
//load another instance of the same person
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
em2.close();
//entity itself was not attached
assertNotSame(person1, person2);
//before merge changes are available; after merge changes are ignored
assertEquals("New Name", person2.getFirstName());
// changed data are NOT available in different entity manager
EntityManager em3 = factory.createEntityManager();
Person person3 = em3.find(Person.class, SIMON_SLASH_ID);
em3.close();
assertNotSame("New Name", person3.getFirstName());
}
</pre><br />
</div></div><a href="" name="EntityManagerUpdates,InsertsandDeletes"></a><h5>Updates, Inserts and Deletes</h5>Modifying data stored in the database with JPA and SQL is similar. You have to open the transaction, perform updates, inserts and deletes and commit the transaction. <br />
<br />
This chapter starts with an overview and short sections on commits, rollbacks and error handling. Each of following sections explains one persistence operation, e.g. update, insert and delete. <br />
<br />
<a href="" name="EntityManagerUpdates,InsertsandDeletesOverview"></a><h6>Overview</h6>Entity manager monitors all changes done on attached entities. If you open and commit the transaction, entity manager will generate SQL queries corresponding to all changes and apply them to the database. <br />
<br />
Of course, only changes done on attached entities are persisted. Detached entities are ignored. <br />
<br />
<a href="" name="EntityManagerUpdates,InsertsandDeletesCommitandRollback"></a><h6>Commit and Rollback</h6>Just as with SQL, commit saves data into the database and rollback throws them away. However, rollback has one important side effect - it detaches all loaded entities. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">/**
* Test the side effect of the rollback method. All
* loaded entities are detached.
*/
@Test
public void entityEquality_rollbacked_transaction() {
EntityManager em = factory.createEntityManager();
//load entity and rollback the transaction
em.getTransaction().begin();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
em.getTransaction().rollback();
//load the same entity again
em.getTransaction().begin();
Person person2 = em.find(Person.class, SIMON_SLASH_ID);
em.getTransaction().commit();
em.close();
//second entity is different from the first one
assertNotSame(person1, person2);
}
</pre><br />
</div></div><a href="" name="EntityManagerUpdates,InsertsandDeletesRuntimeErrors"></a><h6>Exceptions</h6>If the SQL statement causes a database error, for example integrity constraint violation or unique constraint violation, JPA throws runtime exception either during the operation or during the commit. <br />
<br />
You have to handle possible exception in both places.<br />
<br />
<a href="" name="EntityManagerUpdates,InsertsandDeletesUpdate"></a><h6>Update</h6>To update the database, change properties of an attached entity and commit a transaction.<br />
<pre class="brush:java">EntityManager em = factory.createEntityManager();
//open transaction
em.getTransaction().begin();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
//change the entity
person.setFirstName("nobody");
//commit data
em.getTransaction().commit();
</pre><br />
Next example updates a person and saves the change to the database. A different entity manager then loads the same person and checks the change:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void updateEntity() {
EntityManager em1 = factory.createEntityManager();
//open transaction
em1.getTransaction().begin();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
//change the entity
person1.setFirstName("nobody");
//commit data
em1.getTransaction().commit();
em1.close();
//load another entity
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
em2.close();
// entities are different
assertNotSame(person1, person2);
// second entity contains all commited changes
assertEquals("nobody", person2.getFirstName());
}
</pre><br />
</div></div>Note: entity change does not have to happen in between transaction begin and commit. Following works too:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void updateEntity_note() {
EntityManager em1 = factory.createEntityManager();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
//change the entity
person1.setFirstName("hello");
//open transaction and commit data
em1.getTransaction().begin();
em1.getTransaction().commit();
em1.close();
// ... load and check ...
}
</pre><br />
</div></div>Changes on detached entities are ignored. The next test detaches an entity and commits a transaction. Changes are not saved into the database:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void detachedEntity() {
EntityManager em1 = factory.createEntityManager();
//open transaction
em1.getTransaction().begin();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
//change and detach the entity
person1.setFirstName("detacheEntity");
em1.detach(person1);
//commit data
em1.getTransaction().commit();
em1.close();
//load another entity
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
em2.close();
// entities are different
assertNotSame(person1, person2);
// the change was not saved
assertNotSame("detacheEntity", person2.getFirstName());
}
</pre><br />
</div></div><a href="" name="EntityManagerUpdates,InsertsandDeletesInsert"></a><h6>Insert</h6>To insert a new row into the database, a corresponding entity must be introduced to the persistence context. There are two ways how to do it:<br />
<ul><li><code>persist</code> attaches new entity to the entity manager. This method works only if an entity with the same id does not exists,</li>
<li><code>merge</code> adds entity data into the entity manager. The entity is not attached to the entity manager.</li>
</ul><br />
Use the <code>persist</code> method:<br />
<pre class="brush:java">Person newPerson = new Person(3, "BB", "Bob", "Brandert");
EntityManager em = factory.createEntityManager();
em.getTransaction().begin();
//persist the entity
em.persist(newPerson);
em.getTransaction().commit();
</pre><br />
Next test creates a new person, persists it and commits changes. Saved person is loaded with different entity manager and checked:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void insertEntityPersist() {
Person newPerson = new Person(3, "BB", "Bob", "Brandert");
EntityManager em1 = factory.createEntityManager();
em1.getTransaction().begin();
//persist the entity
em1.persist(newPerson);
em1.getTransaction().commit();
//the entity was attached
assertTrue(em1.contains(newPerson));
em1.close();
//new entity was saved and is available to different entity manager
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, 3);
em2.close();
// check loaded entity
assertEquals("BB", person2.getUserName());
}
</pre><br />
</div></div>If an entity with the same id already exists, either persist or a commit method throws an exception:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void insertEntityPersist_Note() {
Person newPerson = new Person(4, "CC", "Cyntia", "Crowd");
EntityManager em1 = factory.createEntityManager();
em1.getTransaction().begin();
//persist the entity
em1.persist(newPerson);
em1.getTransaction().commit();
em1.close();
Person otherPerson = new Person(4, "CC", "Cyntia", "Crowd");
EntityManager em2 = factory.createEntityManager();
em2.getTransaction().begin();
//persist the entity
try {
em2.persist(otherPerson);
em2.getTransaction().commit();
} catch (PersistenceException ex) {
// from the API: The EntityExistsException may be thrown
// when the persist operation is invoked, or the
// EntityExistsException or another PersistenceException
// may be thrown at flush or commit time.
em2.close();
return ;
}
fail("Either persist or commit should throw an exception.");
}
</pre><br />
</div></div>Use the <code>merge</code> method:<br />
<pre class="brush:java">Person newPerson = new Person(2, "MM", "Martin", "Martinez");
EntityManager em = factory.createEntityManager();
em.getTransaction().begin();
//merge entity properties into persistence context
em.merge(newPerson);
em.getTransaction().commit();
</pre><br />
Create a new person, merge it into the persistence context and commit changes. Merged person is loaded with different entity manager and checked:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void insertEntityMerge() {
Person newPerson = new Person(2, "MM", "Martin", "Martinez");
EntityManager em1 = factory.createEntityManager();
em1.getTransaction().begin();
//merge entity properties into persistence context
em1.merge(newPerson);
em1.getTransaction().commit();
em1.close();
//new entity was saved and is available to different entity manager
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, 2);
em2.close();
// check loaded entity
assertEquals("MM", person2.getUserName());
}
</pre><br />
</div></div><a href="" name="EntityManagerUpdates,InsertsandDeletesDelete"></a><h6>Delete</h6>Use the <code>remove</code> method to delete an entity:<br />
<pre class="brush:java">EntityManager em = factory.createEntityManager();
//delete an entity
em.getTransaction().begin();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
em.remove(person1);
em.getTransaction().commit();
</pre><br />
Delete the entity and check whether it was deleted:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void deleteEntity() {
EntityManager em1 = factory.createEntityManager();
//delete an entity
em1.getTransaction().begin();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
em1.remove(person1);
em1.getTransaction().commit();
em1.close();
//try to load deleted entity
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
em2.close();
//entity does not exist anymore
assertNull(person2);
}
</pre><br />
</div></div><a href="" name="EntityManagerRefresh"></a><h5>Refresh</h5>Typically, a thread loads an entity and uses it for a while. If another thread updated the database in between, loaded data are no longer accurate. <br />
<br />
Use the method <code>refresh</code> to update the entity with all changes from the database. Refreshed entity contains only freshly loaded data, e.g. any changes made to the entity are lost. The <code>refresh</code> method can not be used on detached entities.<br />
<pre class="brush:java;">EntityManager em = factory.createEntityManager();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
// ...
em.refresh(person1);
</pre><br />
Refresh test loads an entity with two different entity managers. One updates the entity and another refreshes it. The test checks whether refreshed entity contains changes:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java;">@Test
public void entityEquality_refresh_EntityManager() {
// load an entity
EntityManager em1 = factory.createEntityManager();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
// load the same entity by different entity manager
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
//change the first entity - second entity remains the same
em1.getTransaction().begin();
person1.setFirstName("refreshDemo");
assertNotSame("refreshDemo", person2.getFirstName());
//commit the transaction - second entity still remains the same
em1.getTransaction().commit();
assertNotSame("refreshDemo", person2.getFirstName());
//refresh second entity - it changes
em2.refresh(person2);
assertEquals("refreshDemo", person2.getFirstName());
em1.close();
em2.close();
}
</pre><br />
</div></div>If the other thread deleted the entity, the <code>refresh</code> method throws an <code>EntityNotFoundException</code> exception:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java;">@Test
public void refreshDeletedEntity() {
EntityManager em1 = factory.createEntityManager();
//load the entity
EntityManager em2 = factory.createEntityManager();
Person person2 = em2.find(Person.class, SIMON_SLASH_ID);
//delete it
em1.getTransaction().begin();
Person person1 = em1.find(Person.class, SIMON_SLASH_ID);
em1.remove(person1);
em1.getTransaction().commit();
em1.close();
//refresh loaded entity
try {
em2.refresh(person2);
} catch(EntityNotFoundException ex) {
return ;
} finally {
em2.close();
}
fail("An exception was expected.");
}
</pre><br />
</div></div>The <code>refresh</code> on detached entity throws an exception:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java;">@Test
public void refreshDetachedEntity() {
EntityManager em = factory.createEntityManager();
Person person1 = em.find(Person.class, SIMON_SLASH_ID);
em.detach(person1);
try {
em.refresh(person1);
} catch (IllegalArgumentException ex) {
em.close();
return ;
}
fail("An exception was expected.");
}
</pre><br />
</div></div><a href="" name="Relationships"></a><h4>Relationships</h4>JPA relationship is a reference going from one entity to another entity. In java, relationship is a property that contains entity or collection of entities. In database, relationship corresponds to foreign key between entity tables.<br />
<br />
A Java code with relationship between person and twitter account entities. JPA annotations will be explained later:<br />
<pre class="brush:java;">@Entity
public class TwitterAccount {
@ManyToOne
private Person owner;
}
@Entity
public class Person {
@OneToMany(mappedBy="owner")
private Collection<TwitterAccount> twitterAccounts;
}
</pre><br />
Before we explain which relationship types exist and how to configure them, we have to explain few terms and concepts. Some of them are frequently used in JPA documentation and needed to describe how to configure entity relationships. Others hide little traps for users, so it is better to know about them before you start to code. <br />
<br />
<a href="" name="RelationshipsLazyLoading"></a><h5>Lazy Loading</h5>Most relationships are by default lazy loaded. That means, that data are read from database only if they are needed. </div><pre class="brush:java;">@Test
public void lazyLoading() {
EntityManager em = factory.createEntityManager();
// load the person, twitter accounts are not loaded yet
Person simon = em.find(Person.class, SIMON_SLASH_ID);
// load twitter accounts from the database
Collection<TwitterAccount> accounts = simon.getTwitterAccounts();
assertEquals(2, accounts.size());
// closing entity manager, lazy loading will not be possible anymore
em.close();
}
</pre><div style="text-align: justify;"><br />
This has an important consequence: if you close entity manager or detach the entity, the relationship information will not be available.<br />
<br />
If your relationship returns <code>null</code> (OpenJPA) or throws an exception (Hibernate), check whether entity is still attached to the entity manager and whether entity manager was not closed by mistake. This type of error is pretty common, especially as all examples, including ours, always dutifully close used entity managers.<br />
<br />
Closed entity manager is not able to lazy load data:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java;">@Test
public void lazyLoadingCloseWarning() {
EntityManager em = factory.createEntityManager();
Person simon = em.find(Person.class, SIMON_SLASH_ID);
em.close();
assertNull(simon.getTwitterAccounts());
}
</pre><br />
</div></div><div style="text-align: justify;">Detached entity is not able to lazy load data:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java;">@Test
public void lazyLoadingDetachWarning() {
EntityManager em = factory.createEntityManager();
Person simon = em.find(Person.class, SIMON_SLASH_ID);
em.detach(simon);
assertNull(simon.getTwitterAccounts());
em.close();
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsBidirectionalvsUnidirectional"></a><h5>Bidirectional vs Unidirectional</h5>Any relationship is either 'bidirectional' or 'unidirectional'.<br />
<br />
Lets say the entity A references another entity B. On the java side, the reference B may or may not keep reference back to the entity A. If the entity B references the entity A, the relationship is called 'bidirectional'. If the entity B does not know about the entity A, the relationship is 'unidirectional'.<br />
<br />
Warning: If you use bidirectional relationships, you have to ensure their consistency. If the entity A references the entity B, the entity B must reference back to A. Inconsistent bidirectional relationship may cause unpredictable JPA behavior.<br />
<br />
<a href="" name="RelationshipsBidirectionalvsUnidirectionalSelfConsistentEntities"></a><h6>Self-Consistent Entities</h6>The safest way to achieve the consistency is to create 'smart' setters that always maintain relationship consistency. <br />
<br />
For example, the <code>setOwner</code> method of the <code>TwitterAccount</code> class may automatically remove the twitter account from the old owner and add it to the new owner:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java">/**
* Set new twitter account owner. The method keeps
* relationships consistency:
* * this account is removed from the previous owner
* * this account is added to next owner
*
* @param owner
*/
public void setOwner(Person owner) {
//prevent endless loop
if (sameAsFormer(owner))
return ;
//set new owner
Person oldOwner = this.owner;
this.owner = owner;
//remove from the old owner
if (oldOwner!=null)
oldOwner.removeTwitterAccount(this);
//set myself to new owner
if (owner!=null)
owner.addTwitterAccount(this);
}
private boolean sameAsFormer(Person newOwner) {
return owner==null? newOwner == null : owner.equals(newOwner);
}
</pre><br />
</div></div><div style="text-align: justify;">'Safe' <code>person</code> class on the other side of the relationship would not expose the <code>twitterAccounts</code> collection directly. Instead, it would have specialized <code>addTwitterAccount</code> and <code>removeTwitterAccount</code> methods:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java">/**
* Returns a collection with owned twitter accounts. The
* returned collection is a defensive copy.
*
* @return a collection with owned twitter accounts
*/
public Collection<TwitterAccount> getTwitterAccounts() {
//defensive copy, nobody will be able to change the
//list from the outside
return new ArrayList<TwitterAccountBetter>(twitterAccounts);
}
/**
* Add new account to the person. The method keeps
* relationships consistency:
* * this person is set as the account owner
*/
public void addTwitterAccount(TwitterAccount account) {
//prevent endless loop
if (twitterAccounts.contains(account))
return ;
//add new account
twitterAccounts.add(account);
//set myself into the twitter account
account.setOwner(this);
}
/**
* Removes the account from the person. The method keeps
* relationships consistency:
* * the account will no longer reference this person
* as its owner
*/
public void removeTwitterAccount(TwitterAccount account) {
//prevent endless loop
if (!twitterAccounts.contains(account))
return ;
//remove the account
twitterAccounts.remove(account);
//remove myself from the twitter account
account.setOwner(null);
}
</pre><br />
</div></div><div style="text-align: justify;">Note: most our examples and test cases do not use the safer form. It is mostly because we wanted to keep them as simple as possible. The production code should use the safer version whenever possible.<br />
<br />
<a href="" name="RelationshipsOwningSidevsInverseSide"></a><h5>Owning Side vs Inverse Side</h5>The terms 'owner side' and 'inverse side' are frequently used in JPA documentation and API. They are important because it is impossible to describe how to configure entity relationships without them.<br />
<br />
Entities correspond to database tables and their relationships should correspond to foreign keys between those tables. <br />
<br />
If the tables are joined directly, one of them have column with foreign key pointing to the other entity. The entity stored in the table with the foreign key is called the 'owning side'. The entity referenced by the foreign key is called the 'inverse side'. Unidirectional relationship has only owning side.<br />
<br />
If the tables are joined indirectly through some joining table, any side can be designed as the owning side. It does not matter.<br />
<br />
<a href="" name="RelationshipsPersistingRelationships"></a><h5>Persisting Relationships</h5>JPA assumes that the model is consistent. E.g. entities in bidirectional relationships references each other and deleted entities have been removed from all their relationships. <br />
<br />
If you keep the model consistent, JPA will generally work as intuitively expected. Few less intuitive features are described in following sub-chapters.<br />
<br />
<a href="" name="RelationshipsPersistingRelationshipsRelationshipChanges"></a><h6>Relationship Changes</h6>The relationship between two entities is saved whenever the owner is saved. Saving the inverse does nothing to the relationship, it is only the owner who counts. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">Quote from the specification, page 41: "The owning side of a relationship determines the updates to the relationship in the database."<br />
<br />
Quote from the specification, page 78: "Bidirectional relationships between managed entities will be persisted based on references held by the owning side of the relationship. It is the developer’s responsibility to keep the in-memory references held on the owning side and those held on the inverse side consistent with each other when they change. In the case of unidirectional one-to-one and one-to-many relationships, it is the developer’s responsibility to insure that the semantics of the relationships are adhered to."</div></div>Saving the owner is enough to save the relationship:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void relationshipSaveOnlyOwner_oto() {
EntityManager em = getFactory().createEntityManager();
// load two entities
OneToOneOwner owner = em.find(OneToOneOwner.class, 6);
OneToOneInverse inverse = em.find(OneToOneInverse.class, 6);
// create a relationship between entities
owner.setInverse(inverse);
inverse.setOwner(owner);
// detached inverse will be ignored by commit
em.detach(inverse);
em.getTransaction().begin();
// persist only the owner - it is the only loaded entity
em.getTransaction().commit();
em.close();
// relationship was saved
EntityManager em1 = getFactory().createEntityManager();
OneToOneInverse inverseCheck = em1.find(OneToOneInverse.class, 6);
assertNotNull(inverseCheck.getOwner());
em1.close();
}
</pre><br />
</div></div><div style="text-align: justify;">Saving only the inverse does not save the relationship:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void relationshipSaveOnlyInverse_oto() {
EntityManager em = getFactory().createEntityManager();
// load two entities
OneToOneOwner owner = em.find(OneToOneOwner.class, 7);
OneToOneInverse inverse = em.find(OneToOneInverse.class, 7);
// create a relationship between entities
owner.setInverse(inverse);
inverse.setOwner(owner);
// detached owner will be ignored by commit
em.detach(owner);
em.getTransaction().begin();
// persist only the inverse - it is the only loaded entity
em.getTransaction().commit();
em.close();
// relationship was not saved
EntityManager em1 = getFactory().createEntityManager();
OneToOneOwner ownerCheck = em1.find(OneToOneOwner.class, 7);
assertNull(ownerCheck.getInverse());
em1.close();
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsPersistingRelationshipsEntityDependencies"></a><h6>Entity Dependencies</h6>As the owners table keeps reference to the inverse table, what happen when you persist the owner before the inverse? Will it blindly follow the order in which you wrote those commands and throw an exception on foreign key violation? Will it reorder the statements so that everything works?<br />
<br />
JPA specification does not say how smart an implementation should be. It may do either of these things. That being said, any decent JPA provider is able to automatically resolve entity dependencies. You do not have to persist or delete entities in some special order, JPA will reorder generated SQL statements so that everything works. </div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to Expand</a><br />
<div class="answertext"><br />
<pre class="brush:java">/**
* Any decent JPA provider can be configured to automatically
* resolve relationship dependencies. However, it is not
* required by the specification.
*
* It will reorder those inserts so that no exception is thrown.
*/
@Test
public void wrongOrder_oto() {
// create a relationship between entities
OneToOneOwner owner = new OneToOneOwner(888);
OneToOneInverse inverse = new OneToOneInverse(888);
owner.setInverse(inverse);
inverse.setOwner(owner);
// persist the owner first - JPA is configured to solve dependencies
EntityManager em = getFactory().createEntityManager();
em.getTransaction().begin();
em.persist(owner);
em.persist(inverse);
em.getTransaction().commit();
//check saved data
EntityManager em1 = getFactory().createEntityManager();
OneToOneInverse inverseCheck = em1.find(OneToOneInverse.class, 888);
assertNotNull(inverseCheck.getOwner());
em1.close();
}
</pre><br />
</div></div><div style="text-align: justify;">Hibernate does that automatically, but if you use OpenJPA you have to do a little configuration.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to Expand</a><br />
<div class="answertext"><div style="text-align: justify;">OpenJPA does not correctly resolve SQL statements dependencies because it <a href="https://issues.apache.org/jira/browse/OPENJPA-435">has a bug</a>. OpenJPA by default ignores foreign keys between entity tables. You have two options to make it run. <br />
<br />
First, you can use non-standard annotation <code>ForeignKey</code>. This would make the application non-portable. <br />
<br />
Second, you can configure OpenJPA to read all foreign keys from the database. Place the <code>openjpa.jdbc.SchemaFactory</code> property with <code>native(ForeignKeys=true)</code> value into your persistence.xml:</div><pre class="brush:xml"><persistence-unit name="Cascading" transaction-type="RESOURCE_LOCAL">
...
<properties>
<!-- load foreign key information from the database -->
<property name="openjpa.jdbc.SchemaFactory" value="native(ForeignKeys=true)"/>
</properties>
</persistence-unit>
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsPersistingRelationshipsDeletedEntities"></a><h6>Deleted Entities</h6>If you delete an entity, you have to delete it from all relationships. JPA specification says little about what should happen if you forget to remove it from some of them. <br />
<br />
Of course, if the owner is deleted, its relationship is deleted too. However, if the inverse is deleted, JPA behavior is undefined. Depending on the relationship configuration, the operation either throws an exception or correctly removes both entity and its relationships. What is even worst, if you use legacy database with missing foreign keys, you may end up with an inconsistent database.<br />
<br />
The best is to keep relationships consistent. If you remove an entity, remove it also from all its relationships.<br />
<br />
<a href="" name="RelationshipsPersistingRelationshipsMergingNewEntities"></a><h6>Merging New Entities</h6>As we <a href="#EntityManagerUpdates,InsertsandDeletesInsert">wrote before</a>, you can use <code>merge</code> instead of <code>persist</code> to insert a new entity. This method has one caveat: the merge does not know what to do with a reference to a not-yet-merged entity. Instead of merging it, an exception is thrown:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void impossibleMergeNoCascade() {
// create a relationship between entities
OneToOneOwner owner = new OneToOneOwner(11);
OneToOneInverse inverse = new OneToOneInverse(11);
owner.setInverse(inverse);
inverse.setOwner(owner);
// persist only the owner
EntityManager em = getFactory().createEntityManager();
em.getTransaction().begin();
try {
em.merge(owner);
} catch (RuntimeException ex) {
em.getTransaction().rollback();
}
// check whether previous catch closed the transaction
assertFalse(em.getTransaction().isActive());
em.getTransaction().begin();
try {
em.merge(inverse);
} catch (RuntimeException ex) {
em.getTransaction().rollback();
em.close();
return;
}
fail("It was supposed to throw an exception (twice).");
}
</pre><br />
</div></div>Fortunately, there is a simple workaround. It requires the cascading feature which will be described in the <a href="#RelationshipsCascading">following chapter</a>. In short, if you configure the <code>cascade</code> attribute of the relationship to <code>CascadeType.MERGE</code>, the merge operation will merge both entities at the same time.<br />
<br />
<a href="" name="RelationshipsCascading"></a><h5>Cascading</h5>Unless configured otherwise, whatever you do to an entity happens only to that entity. If you delete an entity, only that entity is deleted. Both entities referenced by it and referencing it remain in database. The same applies for merge, persist, refresh or detach. <br />
<br />
Each relationship can be configured to cascade a selection of these five operations. A cascading operation is applied first to the specified entity. Then it is applied to all referenced entities. If their relationships cascade too, the operation is applied to all entities referenced by them, and so on. <br />
<br />
Cascading helps to solve a lot of relationship dependencies problems. Unfortunately, it is not able to solve all of them and may have some unintended consequences. <br />
<br />
<a href="" name="RelationshipsCascadingConfiguration"></a><h6>Configuration</h6>Each relationship annotation has the <code>cascade</code> attribute. Use it to set a list of cascading operations. Allowed cascading operations are listed in the <code>CascadeType</code> enumeration. It has six members: <code>PERSIST</code>, <code>MERGE</code>, <code>REMOVE</code>, <code>REFRESH</code>, <code>DETACH</code> and <code>ALL</code>.<br />
<br />
For example, following relationship cascades merge and persist operations on one side and all operations on the other side:</div><pre class="brush:java">@Entity
public class CascadeOneToOneOwner {
@OneToOne(cascade=CascadeType.ALL)
private CascadeOneToOneInverse inverse;
}
@Entity
public class CascadeOneToOneInverse {
@OneToOne(mappedBy="inverse", cascade={CascadeType.MERGE, CascadeType.PERSIST})
private CascadeOneToOneOwner owner;
}
</pre><div style="text-align: justify;"><br />
<a href="" name="RelationshipsCascadingLimitation"></a><h6>Limitation</h6>Any cascade type can be used in combination with any relationship with one exception: the <code>REMOVE</code> can be used only with <code>OneToOne</code> and <code>OneToMany</code> relationships.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext">Quote from the specification, page 41: The relationship modeling annotation constrains the use of the <code>cascade=REMOVE</code> specification. The <code>cascade=REMOVE</code> specification should only be applied to associations that are specified as <code>OneToOne</code> or <code>OneToMany</code>. Applications that apply <code>cascade=REMOVE</code> to other associations are not portable.</div></div><a href="" name="RelationshipsCascadingMerge"></a><h6>Merge Problem</h6>Cascading can nicely solve the <a href="#RelationshipsPersistingRelationshipsMergingNewEntities">'merge of new entities with relationship'</a> problem. Recall the problem: if the merged entity references another not-yet-merged entity, the merge operation throws an exception.<br />
<br />
However, the merge works if it cascades through the relationship. JPA implementation will merge both entities within one operation. It will also create a relationship between them. <br />
<br />
Configure the <code>cascade</code> attribute of the relationship to <code>CascadeType.MERGE</code> and the merge will work:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void mergeInverse() {
// create a relationship between entities
CascadeOneToOneOwner owner = new CascadeOneToOneOwner(10);
CascadeOneToOneInverse inverse = new CascadeOneToOneInverse(10);
owner.setInverse(inverse);
inverse.setOwner(owner);
// persist only the inverse
EntityManager em = getFactory().createEntityManager();
em.getTransaction().begin();
em.merge(inverse);
em.getTransaction().commit();
em.close();
// the merge operation was cascaded to the owner too
// the owner was saved to the database
assertEntityExists(CascadeOneToOneOwner.class, 10);
}
@Test
public void mergeOwner() {
// create a relationship between entities
CascadeOneToOneOwner owner = new CascadeOneToOneOwner(12);
CascadeOneToOneInverse inverse = new CascadeOneToOneInverse(12);
owner.setInverse(inverse);
inverse.setOwner(owner);
// persist only the owner
EntityManager em = getFactory().createEntityManager();
em.getTransaction().begin();
em.merge(owner);
em.getTransaction().commit();
// the merge operation was cascaded to the inverse too
// the inverse was saved to the database
assertEntityExists(CascadeOneToOneInverse.class, 12);
}
</pre><br />
</div></div><a href="" name="RelationshipsCascadingCaution"></a><h6>Caution</h6>It might be tempting to cascade all operations on all relationships. However, that is a bad idea for two reasons. First, that would badly affect the performance. Each operation would have potential to cascade to very large number of objects, potentially whole object graph.<br />
<br />
The cascading of <code>remove</code> and <code>persist</code> operations applies also on those entities that have not been loaded yet. It even passes through them to other entities, potentially traversing through whole object graph.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">The class <code>CascadingTestCase</code> located <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/test/java/org/meri/jpa/cascading/CascadingTestCase.java">on Github</a> contains two tests proving this claim. <br />
<br />
Both tests assume that database contains a three etities: <code>first</code>, <code>second</code> and <code>third</code>. They reference each other via cascading relationships and all three have the same id.<br />
<br />
Proof that delete cascades through unloaded entities:<br />
<pre class="brush:java">@Test
public void cascadeDeleteThroughUnloaded() {
// check the data: first, second and third entities are
// chained and have the same id
checkRemoveChain(1);
// remove first entity
EntityManager em = getFactory().createEntityManager();
em.getTransaction().begin();
CascadeRemoveFirst first = em.find(CascadeRemoveFirst.class, 1);
em.remove(first);
em.getTransaction().commit();
em.close();
// both second and third entities have been removed
// Note: neither one was loaded by the original entity manager
assertEntityNOTExists(CascadeRemoveSecond.class, 1);
assertEntityNOTExists(CascadeRemoveThird.class, 1);
}
</pre><br />
A test proving that persist cascades through unloaded entities is named <code>cascadePersistThroughUnloaded</code>. It is available in the same class.</div></div>The cascading of <code>refresh</code>, <code>merge</code> and <code>detach</code> passes only through entities that are already loaded.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">The class <code>CascadingTestCase</code> located <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/test/java/org/meri/jpa/cascading/CascadingTestCase.java">on Github</a> contains three tests proving this claim. <br />
<br />
All three tests assume that database contains a three etities: <code>first</code>, <code>second</code> and <code>third</code>. They reference each other via cascading relationships and all three have the same id.<br />
<br />
Proof that detach does not cascade through unloaded entities:<br />
<pre class="brush:java">@Test
public void cascadeDetachThroughUnloaded() {
// check the data: first, second and third entities are
// chained and have the same id
checkChain(2);
EntityManager em = getFactory().createEntityManager();
// load both first and third element of chain
CascadeFirst first = em.find(CascadeFirst.class, 2);
CascadeThird third = em.find(CascadeThird.class, 2);
// detach first element
em.detach(first);
// only first element was detached
// detach operation does not cascade through unloaded elements
assertFalse(em.contains(first));
assertTrue(em.contains(third));
em.close();
}
</pre><br />
A tests proving that merge and refrest do not cascade through unloaded entities are named <code>cascadeMergeThroughUnloaded</code> and <code>cascadeRefreshThroughUnloaded</code>. Both are available in the same class.</div></div>Second, the operation might cascade to unintended entities. For example, you might remove one entity and persist another. If the persist operation cascades too far, it will resurrect removed entity back to life.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext">Resurrect removed entity test:<br />
<pre class="brush:java">@Test
public void ressurectDeletedEntity() {
EntityManager em = getFactory().createEntityManager();
em.getTransaction().begin();
// load two related entities
CascadeOneToOneOwner owner = em.find(CascadeOneToOneOwner.class, 2);
CascadeOneToOneInverse inverse = owner.getInverse();
//delete one of them
em.remove(owner);
//new entity demo references second one
CascadePersistDemo demo = new CascadePersistDemo(2);
demo.setInverse(inverse);
//persist the demo
em.persist(demo);
em.getTransaction().commit();
em.close();
//the persist operation cascaded through the 'inverse'
//to the removed 'owner'. The owner was not deleted.
assertEntityExists(CascadeOneToOneOwner.class, 2);
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsConfigurationOverview"></a><h5>Configuration Overview</h5>Each relationship property must be marked by one of <code>@OneToOne</code>, <code>@OneToMany</code>, <code>@ManyToOne</code> and <code>@ManyToMany</code> annotations. Each of these four annotations define defaults for underlying database structure, so theoretically no other configuration is needed. <br />
<br />
Of course, database columns, tables and foreign keys are configurable. If you are not satisfied with defaults, use one of <code>@JoinColumn</code>, <code>@JoinColumns</code>, <code>@JoinTable</code>, <code>@PrimaryKeyJoinColumn</code> and <code>@PrimaryKeyJoinColumns</code> annotations to customize. These configuration annotations are allowed only on the 'owning side'. If you try to use them on the 'inverse side', JPA will throw an exception.<br />
<br />
Each sub-chapter of this chapter contain short overview on one of relationship annotations. The overview always describes the relationship type and contains an example. All other details are explained in following chapters.<br />
<br />
<a href="" name="RelationshipsConfigurationOverviewOneToOne"></a><h6>OneToOne</h6>Each annotated entity references only one entity. Each opposite side entity is referenced by one owning entity. <br />
<br />
Example: A person may have only one Facebook account and each Facebook account is owned by one person.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java;">@Entity
public class FacebookAccount {
@OneToOne
private Person owner;
}
@Entity
public class Person {
@OneToOne(mappedBy="owner")
private FacebookAccount facebookAccount;
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsConfigurationOverviewOneToMany"></a><h6>OneToMany</h6>Each annotated entity can reference multiple entities. However, the opposite side entity may NOT be referenced by multiple entities.<br />
<br />
Example: A person may have any number of Twitter accounts. However, each Twitter account is owned by one person.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java;">@Entity
public class TwitterAccount {
@ManyToOne
private Person owner;
}
@Entity
public class Person {
@OneToMany(mappedBy="owner")
private Collection<TwitterAccount> twitterAccounts;
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsConfigurationOverviewManyToOne"></a><h6>ManyToOne</h6>The opposite of one-to-many relationship. The annotated entity can reference only one entity. However, any opposite side entity may be referenced by multiple entities.<br />
<br />
Example: Each Twitter account is owned by one person. However, a person may have any number of Twitter accounts. </div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java;">@Entity
public class TwitterAccount {
@ManyToOne
private Person owner;
}
@Entity
public class Person {
@OneToMany(mappedBy="owner")
private Collection<TwitterAccount> twitterAccounts;
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsConfigurationOverviewManyToMany"></a><h6>ManyToMany</h6>Each annotated entity references multiple entities. Each opposite side entity can be referenced by any number of annotated entities. <br />
<br />
Example: Each person may follow any number of twitter accounts. Each twitter account may be followed by any number of people.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java;">@Entity
public class TwitterAccount {
@ManyToMany
private Set<Person> followers;
}
@Entity
public class Person {
@ManyToMany(mappedBy="followers")
@MapKey(name="accountName")
private Map<String, TwitterAccount> twitterFollowing;
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsOneToOne"></a><h5>One-To-One</h5>Two entity classes have one-to-one relationship if each entity on one side corresponds to one entity on the other side. <br />
<br />
For example, a person and its Facebook account have one-to-one relationship. The person may have only one Facebook account and each Facebook account is owned by one person.<br />
<br />
<a href="" name="RelationshipsOneToOneBasicConfiguration"></a><h6>Basic Configuration</h6>To create bidirectional relationship, place the <code>@OneToOne</code> annotation on both sides of the relationship. Decide where you want to keep the joining column. That side is going to be the owner.<br />
<br />
The inverse side must specify the <code>mappedBy</code> annotation attribute. If it is missing, JPA treats the relationship as two separate unidirectional relationships. The attribute contains the name of the owners property that keeps the relationship.<br />
<br />
The owner side can not have the <code>mappedBy</code> attribute specified. <br />
<pre class="brush:java;">@Entity
public class OneToOneOwner {
@OneToOne
private OneToOneInverse inverse;
}
@Entity
public class OneToOneInverse {
@OneToOne(mappedBy="inverse")
private OneToOneOwner owner;
}
</pre><br />
To create unidirectional relationship, place the <code>@OneToOne</code> annotation on the only side of the relationship. This side is going to be the owner.<br />
<br />
<a href="" name="RelationshipsOneToOneDefaultDatabaseStructure"></a><h6>Default Database Structure</h6>Unless configured otherwise, JPA assumes that the owner table contains a column named <code>PROPERTYNAME_ID</code>. This column should contain references to the inverse side entity.<br />
<br />
The previous relationship corresponds to following db structure:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:sql">CREATE TABLE onetooneowner (
id INT NOT NULL,
inverse_id INT,
CONSTRAINT PK_ONETOONEOWNER PRIMARY KEY (id), UNIQUE (inverse_id)
);
CREATE TABLE onetooneinverse (
id INT NOT NULL,
CONSTRAINT PK_ONETOONEINVERSE PRIMARY KEY (id)
);
ALTER TABLE onetooneowner
ADD CONSTRAINT fk_onetooneownerinverse FOREIGN KEY (inverse_id)
REFERENCES onetooneinverse (id);
</pre><br />
</div></div><a href="" name="RelationshipsOneToOneOptional"></a><h6>Optional</h6>If the annotation attribute <code>optional</code> is set to <code>false</code>, the relationship is mandatory. The entity must reference another entity. Relationship property can not be <code>null</code>. <br />
<br />
If you try to persist an entity with empty mandatory property, JPA will throw an undocumented runtime exception.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void mandatoryOneToOne_inverse() {
EntityManager em = factory.createEntityManager();
em.getTransaction().begin();
LazyOneToOneInverse inverse = new LazyOneToOneInverse(10);
try {
em.persist(inverse);
em.getTransaction().commit();
} catch (RuntimeException ex) {
em.close();
return ;
}
fail("The transaction was supposed to fail.");
}
</pre><br />
Owner with mandatory relationship is tested in similar test named <code>mandatoryOneToOne_owner</code>. It is defined in <code>OneToOneTestCase</code> test case available <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/test/java/org/meri/jpa/relationships/OneToOneTestCase.java#L74">on Github</a>.</div></div><a href="" name="RelationshipsOneToOneLoading"></a><h6>Loading</h6>One-to-one relationship is eagerly loaded. Lazy loading is possible only on the owner side and only if the relationship is not optional, e.g. the <code>optional</code> annotation attribute is set to <code>false</code>.<br />
<br />
Lazily loading owner: <br />
<pre class="brush:java">@Entity
public class LazyOneToOneOwner {
@OneToOne(optional=false,fetch=FetchType.LAZY)
private LazyOneToOneInverse inverse;
}
</pre><br />
Test verifying the owner:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void lazyOneToOne_owner() {
EntityManager em = factory.createEntityManager();
LazyOneToOneOwner owner = em.find(LazyOneToOneOwner.class, 1);
em.close();
//The relation ship is lazily loaded and the manager is already closed
assertNull(owner.getInverse());
}
</pre><br />
</div></div>The inverse side was set to lazy too:<br />
<pre class="brush:java">public class LazyOneToOneInverse {
//lazy loading on the inverse side DOES NOT WORK
//the parameter fetch=FetchType.LAZY is useless here
@OneToOne(mappedBy="inverse",optional=false,fetch=FetchType.LAZY)
private LazyOneToOneOwner owner;
}
</pre><br />
This side of the relationship is still eagerly loaded:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void lazyOneToOne_inverse() {
EntityManager em = factory.createEntityManager();
OneToOneInverse owner = em.find(OneToOneInverse.class, 5);
em.close();
//lazy loading is possible only on the owner side
assertEquals(1, owner.getOwner().getId());
}
</pre><br />
</div></div><a href="" name="RelationshipsOneToOneCascading"></a><h6>Cascading</h6>All cascading types are compatible with one-to-one relationship.<br />
<br />
<a href="" name="RelationshipsOneToOneConsistency"></a><h6>Consistency</h6>If the relationship is bidirectional, you are responsible for keeping it consistent. The safe way how to do it, is to let setters keep their own consistency.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">One possible way how to write safe setters is available in <a href="href="https://github.com/SomMeri/org.meri.jpa.tutorial/tree/master/src/main/java/org/meri/jpa/relationships/entities/bestpractice"><code>org.meri.jpa.relationships.entities.bestpractice</code></a> package in our demo project. Related test case is in class <code>SafeRelationshipsTestCase</code>.<br />
<br />
<pre class="brush:java">@Entity
public class Person {
@OneToOne(mappedBy = "owner")
private FacebookAccount facebookAccount;
public void setFacebookAccount(FacebookAccount facebookAccount) {
//prevent endless loop
if (sameAsFormer(facebookAccount))
return;
//set new facebook account
FacebookAccount oldAccount = this.facebookAccount;
this.facebookAccount = facebookAccount;
//remove from the old facebook account
if (oldAccount!=null)
oldAccount.setOwner(null);
//set myself into new facebook account
if (facebookAccount!=null)
facebookAccount.setOwner(this);
}
private boolean sameAsFormer(SafeFacebookAccount newAccount) {
return facebookAccount == null ?
newAccount == null : facebookAccount.equals(newAccount);
}
}
@Entity
public class FacebookAccount {
// ... analogique to the person ...
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsOneToOneCustomJoinColumn"></a><h6>Custom Join Column</h6>Use the annotation <code>@JoinColumn</code> if you wish to customize database column name. This annotation must be used on the owner side. <br />
<br />
The following example uses a <code>customcolumn</code> database column instead of the default one:<br />
<pre class="brush:java">@Entity
public class ColumnOneToOneOwner {
@OneToOne
@JoinColumn(name="customcolumn")
private ColumnOneToOneInverse inverse;
}
@Entity
public class ColumnOneToOneInverse {
@Id
@Column(name="inverse_id")
private long id;
@OneToOne(mappedBy="inverse")
private ColumnOneToOneOwner owner;
}
</pre>Note: the <code>@JoinColumn</code> annotation has <code>referencedColumnName</code> attribute. It is not needed in this context. JPA will use id column of the inverse entity, even if it was renamed. <br />
<br />
Coresponding database structure:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext"><br />
<pre class="brush:sql">CREATE TABLE onetooneowner (
id INT NOT NULL,
customcolumn INT,
CONSTRAINT
PK_ONETOONEOWNER PRIMARY KEY (id),
UNIQUE (customcolumn)
);
CREATE TABLE onetooneinverse (
inverse_id INT NOT NULL,
CONSTRAINT PK_ONETOONEINVERSE PRIMARY KEY (inverse_id)
);
ALTER TABLE onetooneowner
ADD CONSTRAINT fk_onetooneownerinverse FOREIGN KEY (customcolumn)
REFERENCES onetooneinverse (inverse_id);
</pre><br />
</div></div>Warning: the annotation <code>@JoinColumn</code> has also a <code>table</code> attribute. This attribute is incompatible with one-to-one relationship. Theoretically, it should contain the name of the table that contains the column. We tried it twice and each attempt resulted in a weird SQL when we tried to load an entity.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">First attempt switched owner and inverse tables: <br />
<pre class="brush:java">@JoinColumn(name="customcolumn", table="ColumnOneToOneInverse")</pre>Resulting load owner SQL ignored the <code>customcolumn</code> column:<br />
<pre class="brush:sql">SELECT t1.id, t2.id
FROM ColumnOneToOneOwner t0
INNER JOIN ColumnOneToOneInverse t1 ON t0.INVERSE_ID = t1.id
LEFT OUTER JOIN ColumnOneToOneOwner t2 ON t1.id = t2.INVERSE_ID
</pre><br />
Second attempt introduced a new table 'bumbum': <br />
<pre class="brush:java">@JoinColumn(name="customcolumn", table="bumbum")</pre>JPA produced SQL that does full join on the owner table:<br />
<pre class="brush:sql">SELECT t1.id, t3.id
FROM bumbum t0
INNER JOIN ColumnOneToOneInverse t1 ON t0.INVERSE_ID = t1.id
LEFT OUTER JOIN bumbum t2 ON t1.id = t2.INVERSE_ID
, ColumnOneToOneOwner t3
WHERE t0.OWNER_ID = ?
</pre><br />
</div></div><a href="" name="RelationshipsOneToOneCustomJoinTable"></a><h6>Custom Join Table</h6>It is not possible to implement one-to-one relationship between two entities with a joining table. First of all, it is neither a good design nor a good idea. If you have a legacy database or have no say over database structure, you will have to use other then one-to-one relationship type in your model.<br />
<br />
It might be tempting to use either <code>@JoinTable</code> annotation or the <code>table</code> attribute of the <code>@JoinColumn</code> annotation. Neither one works.<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">According to JPA API, the <code>@JoinTable</code> annotation should be placed on the owning side of a many-to-many association, or in an unidirectional one-to-many association. One-to-one relationship is not listed. We tried it anyway, just to see whether the thing works.<br />
<br />
The test method <code>insertUpdateLoadTableOneToOne</code> is available in the <code>OneToOneTestCase</code> class <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/test/java/org/meri/jpa/relationships/OneToOneTestCase.java#L159">on Github</a>.<br />
<br />
Insert, update and load of the owner entity work correctly. However, load of the inverse entity does not work. It generates three selects and two of them perform full join of two tables:<br />
<pre class="brush:sql">// this is OK
SELECT t0.inverse_id FROM TableOneToOneInverse t0 WHERE t0.inverse_id = ?
//full join!!
SELECT t1.owner_id FROM TableOneToOneJoin t0, TableOneToOneOwner t1
WHERE t0.join_inverse_id = ?
// another full join
SELECT t1.inverse_id, t3.owner_id
FROM TableOneToOneJoin t0
INNER JOIN TableOneToOneInverse t1 ON t0.join_inverse_id = t1.inverse_id
LEFT OUTER JOIN TableOneToOneJoin t2 ON t1.inverse_id = t2.join_inverse_id, TableOneToOneOwner t3
WHERE t0.join_owner_id = ?
</pre><br />
</div></div><a href="" name="RelationshipsOneToOneJoinTroughPrimaryKey"></a><h6>Join Trough Primary Key</h6>Finally, it is possible to join entities through their primary keys. There is no special join column, entities are in relationship if their primary keys are equal. Place <code>@PrimaryKeyJoinColumn</code> annotation on the owner side:<br />
<pre class="brush:java">@Entity
public class PrimaryOneToOneOwner {
@OneToOne
@PrimaryKeyJoinColumn
private PrimaryOneToOneInverse inverse;
}
@Entity
public class PrimaryOneToOneInverse {
@OneToOne(mappedBy="inverse")
private PrimaryOneToOneOwner owner;
}
</pre><br />
Corresponding database structure:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:sql">CREATE TABLE primaryonetooneowner (
id INT NOT NULL,
CONSTRAINT PK_PRIMARYONETOONEOWNER PRIMARY KEY (id)
);
CREATE TABLE primaryonetooneinverse (
id INT NOT NULL,
CONSTRAINT PK_PRIMARYONETOONEINVERSE PRIMARY KEY (id)
);
ALTER TABLE primaryonetooneowner
ADD CONSTRAINT fk_primaryonetooneownerinverse FOREIGN KEY (id)
REFERENCES primaryonetooneinverse (id);
</pre><br />
</div></div><a href="" name="RelationshipsBidirectionalOneToManyManyToOne"></a><h5>Bidirectional One-To-Many / Many-To-One</h5>One-to-many and many-to-one relationships are opposites of each other. If one side of the relationship is one-to-many, the other side must be many-to-one. Of course, the same goes the other way round. The opposite of many-to-one must be one-to-many.<br />
<br />
The one-to-many side is able to reference any number of entities and keeps them in collection or in map. However, each instance in the many-to-one side can reference only one entity.<br />
<br />
For example, a person and its Twitter accounts have this type of relationship. The person may have any number of Twitter accounts. Each Twitter account is owned by one person.<br />
<br />
<a href="" name="RelationshipsBidirectionalOneToManyManyToOneBasicConfiguration"></a><h6>Basic Configuration</h6>The property referencing multiple entities must be a <code>Collection</code>, a <code>List</code>, a <code>Set</code> or a <code>Map</code>. No other type is allowed. Mark it with <code>@OneToMany</code> annotation. This is the inverse side and must specify the <code>mappedBy</code> annotation attribute. <br />
<br />
The side referencing only one entity is the owner and must be annotated with the <code>@ManyToOne</code> annotation.<br />
<br />
If the inverse side property is either a <code>List</code>, a <code>Set</code> or a <code>Collection</code>, nothing else is needed.<br />
<pre class="brush:java">@Entity
public class CollectionOwner {
@ManyToOne
private CollectionInverse inverse;
}
@Entity
public class CollectionInverse {
@OneToMany(mappedBy="inverse")
private Collection<CollectionOwner> owners;
}
</pre><br />
If the inverse side property is a <code>Map</code>, the owner must have a property which contains a map key. Place also <code>@MapKey</code> annotation on the map. Annotation attribute <code>name</code> must contain the name of the property with a key:<br />
<pre class="brush:java">@Entity
public class MapOwner {
@Id
private long id;
private String mapKey;
@ManyToOne
private MapInverse inverse;
}
@Entity
public class MapInverse {
@OneToMany(mappedBy="inverse")
@MapKey(name="mapKey")
private Map<String, MapOwner> owners;
}
</pre><br />
The property specified in <code>@MapKey</code> will be used as a key to the map:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void basicMap() {
EntityManager em = factory.createEntityManager();
MapOwner owner = em.find(MapOwner.class, 1);
MapInverse inverse = owner.getInverse();
// mapKey property is a key to the map
Map<String, MapOwner> owners = inverse.getOwners();
assertEquals(owner, owners.get(owner.getMapKey()));
em.close();
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsBidirectionalOneToManyManyToOneDefaultDatabaseStructure"></a><h6>Default Database Structure</h6>Unless configured otherwise, JPA assumes that the owners table, e.g. the one with entities on <code>ManyToOne</code> side, contains a column named <code>PROPERTYNAME_ID</code>. This column should contain references to the inverse side entity. <br />
<br />
If a <code>Map</code> is used, the owner table must contain also a column corresponding to the map key property. The map key property should be unique.<br />
<br />
Db structure for a version with a collection:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:sql">CREATE TABLE collectionowner (
id INT NOT NULL,
inverse_id INT,
CONSTRAINT PK_COLLECTIONOWNER PRIMARY KEY (id)
);
CREATE TABLE collectioninverse (
id INT NOT NULL,
CONSTRAINT PK_COLLECTIONINVERSE PRIMARY KEY (id)
);
ALTER TABLE collectionowner
ADD CONSTRAINT fk_collectionownerinverse FOREIGN KEY (inverse_id)
REFERENCES collectioninverse (id);
</pre><br />
</div></div>Db structure for a version with a map:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:sql">CREATE TABLE mapowner (
id INT NOT NULL,
mapkey VARCHAR(1500) NOT NULL,
inverse_id INT,
CONSTRAINT PK_MAPOWNER PRIMARY KEY (id),
UNIQUE (mapkey)
);
CREATE TABLE mapinverse (
id INT NOT NULL,
CONSTRAINT PK_MAPINVERSE PRIMARY KEY (id)
);
ALTER TABLE mapowner
ADD CONSTRAINT fk_mapownerinverse FOREIGN KEY (inverse_id)
REFERENCES mapinverse (id);
</pre><br />
</div></div><a href="" name="RelationshipsBidirectionalOneToManyManyToOneLoading"></a><h6>Loading </h6>The owner, e.g. <code>@ManyToOne</code> side of the relationship, is eagerly loading by default. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void eagerLoadingCollection() {
EntityManager em = factory.createEntityManager();
CollectionOwner owner = em.find(CollectionOwner.class, 1);
em.close();
//this side is eagerly loaded by default
CollectionInverse inverse = owner.getInverse();
assertEquals(5, inverse.getId());
}
</pre><br />
</div></div>Unlike in <a href="#RelationshipsOneToOne">one-to-one relationship</a>, it is possible to configure it to be lazy:<br />
<pre class="brush:java">@ManyToOne(fetch=FetchType.LAZY)
private LazyInverse inverse;
</pre><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Test
public void lazyLoadingCollection() {
EntityManager em = factory.createEntityManager();
LazyOwner owner = em.find(LazyOwner.class, 1);
em.close();
// it is possible to configure it to be lazy
assertNull(owner.getInverse());
}
</pre><br />
</div></div><a href="" name="RelationshipsBidirectionalOneToManyManyToOneCascading"></a><h6>Cascading</h6>All cascading types are compatible with one-to-many side of the relationship. The opposite many-to-one side does not support the <code>REMOVE</code> cascade type. It supports only <code>PERSIST</code>, <code>MERGE</code>, <code>REFRESH</code> and <code>DETACH</code>.<br />
<br />
<a href="" name="RelationshipsBidirectionalOneToManyManyToOneConsistency"></a><h6>Consistency</h6>You are responsible for keeping the relationship consistent. The safe way how to do it, is to hide the collection or map and expose only safe consistency keeping methods.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">One possible way how to write safe accessor methods is available in <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/tree/master/src/main/java/org/meri/jpa/relationships/entities/bestpractice"><code>org.meri.jpa.relationships.entities.bestpractice</code></a> package in our demo project. Related test case is named <code>SafeRelationshipsTestCase</code>.<br />
<br />
The example with a collection:<br />
<pre class="brush:java">@Entity
public class Person {
@OneToMany(mappedBy = "owner")
private Collection<TwitterAccount> twitterAccounts =
new ArrayList<SafeTwitterAccount>();
/**
* Returns a collection with owned twitter accounts. The
* returned collection is a defensive copy.
*
* @return a collection with owned twitter accounts
*/
public Collection<TwitterAccount> getTwitterAccounts() {
//defensive copy, nobody will be able to change
//the list from the outside
return new ArrayList<TwitterAccount>(twitterAccounts);
}
/**
* Add new account to the person. The method keeps
* relationships consistency:
* * this person is set as the account owner
*/
public void addTwitterAccount(TwitterAccount account) {
//prevent endless loop
if (twitterAccounts.contains(account))
return ;
//add new account
twitterAccounts.add(account);
//set myself into the twitter account
account.setOwner(this);
}
/**
* Removes the account from the person. The method keeps
* relationships consistency:
* * the account will no longer reference this person as its owner
*/
public void removeTwitterAccount(TwitterAccount account) {
//prevent endless loop
if (!twitterAccounts.contains(account))
return ;
//remove the account
twitterAccounts.remove(account);
//remove myself from the twitter account
account.setOwner(null);
}
}
@Entity
public class FacebookAccount {
/**
* Set new twitter account owner. The method keeps
* relationships consistency:
* * this account is removed from the previous owner
* * this account is added to next owner
*
* @param owner
*/
public void setOwner(SafePerson owner) {
//prevent endless loop
if (sameAsFormer(owner))
return ;
//set new owner
SafePerson oldOwner = this.owner;
this.owner = owner;
//remove from the old owner
if (oldOwner!=null)
oldOwner.removeTwitterAccount(this);
//set myself to new owner
if (owner!=null)
owner.addTwitterAccount(this);
}
private boolean sameAsFormer(SafePerson newOwner) {
return owner==null? newOwner == null : owner.equals(newOwner);
}
}
</pre><br />
</div></div><div style="text-align: justify;">Additionally, if you used a map, be careful about changing the value of the map key property.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><div style="text-align: justify;">The next example shows one possible way how to write safe map key setter. It assumes that the opposite entity keeps twitter accounts in a map instead of collection. The <code>accountName</code> is the key to the map.</div><pre class="brush:java""">public void setAccountName(String accountName) {
SafePerson myOwner = owner;
myOwner.removeTwitterAccount(this);
this.accountName = accountName;
myOwner.addTwitterAccount(this);
}
</pre><br />
This method has one drawback: depending on the JPA implementation, it may cause unnecessary database calls. </div></div><div style="text-align: justify;"><a href="" name="RelationshipsBidirectionalOneToManyManyToOneOptional"></a><h6>Optional</h6>The <code>@ManyToOne</code> annotation has the attribute <code>optional</code>. If it is set to false, the relationship is mandatory. The entity must reference another entity. Relationship property can not be <code>null</code>.<br />
<br />
If you try to persist an entity with empty mandatory property, JPA will throw an undocumented runtime exception.<br />
<br />
The opposite <code>@OneToMany</code> annotation does not have such attribute.<br />
<br />
<a href="" name="RelationshipsBidirectionalOneToManyManyToOneOrphanRemoval"></a><h6>Orphan Removal</h6>An orphan in a bidirectional one-to-many many-to-one relationship is an owner entity that does not belong to the collection or map on the inverse side. In other words, an orphan is entity on the many-to-one side that does not reference an entity on the one-to-many side.<br />
<br />
If the <code>orphanRemoval</code> attribute of <code>@OneToMany</code> annotation is set to true, JPA will automatically delete orphans. If you remove an entity from the annotated collection or map, JPA will delete it from the database.<br />
<br />
Set orphan removal:</div><pre class="brush:java">@OneToMany(mappedBy="inverse", orphanRemoval=true)
@MapKey(name="mapKey")
private Map<String, OrphanOwner> owners;
</pre><div style="text-align: justify;"><br />
Delete the 'first' entity:</div><pre class="brush:java">// load the inverse
OrphanInverse inverse = em.find(OrphanInverse.class, 5);
// remove the 'first' owner
inverse.getOwners().remove("first");
// commit the transaction and close manager
em.getTransaction().commit();
</pre><div style="text-align: justify;"><br />
Expand to see full orphan removal test:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void orphanRemoval() {
EntityManager em = factory.createEntityManager();
em.getTransaction().begin();
// load the inverse
OrphanInverse inverse = em.find(OrphanInverse.class, 5);
// check whether the owner exists
assertEquals(1, inverse.getOwners().get("first").getId());
// remove the owner
inverse.getOwners().remove("first");
// commit the transaction and close manager
em.getTransaction().commit();
em.close();
// owner without inverse has been removed
assertEntityNOTExists(OrphanOwner.class, 1);
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsBidirectionalOneToManyManyToOneOrdering"></a><h6>Ordering</h6>If the one-to-many side of the relationship is a list, it may be sorted with the <code>@OrderBy</code> annotation. If you put property name followed by <code>ASC</code> or <code>DESC</code> into its value, JPA will use it to sort loaded entities. If you need multiple sorting properties, separate them with comma ','.<br />
<br />
Sort by the <code>ordering</code> property. Entities with equal <code>ordering</code> property are sorted by <code>id</code>:<br />
<pre class="brush:java">@Entity
public class OrderedInverse {
@Id
private long id;
@OneToMany(mappedBy="inverse")
@OrderBy("ordering ASC, id DESC")
private List<OrderedOwner> owners;
}
@Entity
public class OrderedOwner {
@Id
private long id;
@Column(name="orderby")
private long ordering;
@ManyToOne
private OrderedInverse inverse;
}
</pre><br />
The test of ordering is available in <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/blob/master/src/test/java/org/meri/jpa/relationships/OneToMany_ManyToOneTestCase.java#L136"><code>OneToMany_ManyToOneTestCase</code></a> test case. <br />
<br />
<a href="" name="RelationshipsBidirectionalOneToManyManyToOneCustomJoinColumn"></a><h6>Custom Columns</h6>Use the annotation <code>@JoinColumn</code> if you wish to customize the join column name. Map key column name is customized using the <code>@Column</code> annotations.<br />
<br />
Following example uses a <code>customcolumn</code> database column instead of the default one:<br />
<pre class="brush:java">@Entity
public class CustomColumnCollectionOwner {
private String mapKey;
@ManyToOne
@JoinColumn(name="customcolumn")
private CustomColumnCollectionInverse inverse;
}
@Entity
public class CustomColumnCollectionInverse {
@OneToMany(mappedBy="inverse")
private Collection<CustomColumnCollectionOwner> owners;
}
</pre><br />
Coresponding database structure:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:sql">CREATE TABLE customcolumncollectionowner (
id INT NOT NULL,
customcolumn INT,
CONSTRAINT PK_CUSTOMCOLUMNCOLLECTIONOWNER PRIMARY KEY (id)
);
CREATE TABLE customcolumncollectioninverse (
id INT NOT NULL,
CONSTRAINT PK_CUSTOMCOLUMNCOLLECTIONINVERSE PRIMARY KEY (id));
ALTER TABLE customcolumncollectionowner
ADD CONSTRAINT fk_customcolumncollectionownerinverse FOREIGN KEY (customcolumn)
REFERENCES customcolumncollectioninverse (id);
</pre><br />
</div></div>Next example uses a <code>customcolumn</code> and <code>customMapKey</code> database columns instead of default ones:<br />
<pre class="brush:java">@Entity
public class CustomColumnMapOwner {
@Column(name="customMapKey")
private String mapKey;
@ManyToOne
@JoinColumn(name="customcolumn")
private CustomColumnMapInverse inverse;
}
@Entity
public class CustomColumnMapInverse {
@OneToMany(mappedBy="inverse")
@MapKey(name="mapKey")
private Map<String, CustomColumnMapOwner> owners;
}
</pre><br />
Coresponding database structure:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext"><br />
<pre class="brush:sql">CREATE TABLE customcolumnmapowner (
id INT NOT NULL,
custommapkey VARCHAR(1500) NOT NULL,
customcolumn INT,
CONSTRAINT PK_CUSTOMCOLUMNMAPOWNER PRIMARY KEY (id),
UNIQUE (custommapkey)
);
CREATE TABLE customcolumnmapinverse (
id INT NOT NULL,
CONSTRAINT PK_CUSTOMCOLUMNMAPINVERSE PRIMARY KEY (id)
);
ALTER TABLE customcolumnmapowner
ADD CONSTRAINT fk_customcolumnmapownerinverse FOREIGN KEY (customcolumn)
REFERENCES customcolumnmapinverse (id);
</pre><br />
</div></div><div style="text-align: justify;">Note: the <code>@JoinColumn</code> annotation has <code>referencedColumnName</code> attribute. It is not needed in this context. JPA will use id column of the inverse entity. <br />
<br />
Warning: the annotation <code>@JoinColumn</code> has also a <code>table</code> attribute. This attribute is incompatible with many-to-one relationship. Do not use it.<br />
<br />
<a href="" name="RelationshipsBidirectionalOneToManyManyToOneCustomJoinTable"></a><h6>Custom Join Table</h6>It is not possible to implement bidirectional one-to-many many-to-one relationship between two entities with a joining table. First of all, it is neither a good design nor a good idea. If you have a legacy database or have no say over database structure, you will have to use other relationship type in your model.<br />
<br />
It might be tempting to use either <code>@JoinTable</code> annotation or the table attribute of the <code>@JoinColumn</code> annotation. Neither one works.<br />
<br />
<a href="" name="RelationshipsUnidirectionalManyToOne"></a><h5>Unidirectional Many-To-One</h5>Unidirectional many-to-one relationship is exactly the same as the <a href="#RelationshipsBidirectionalOneToManyManyToOne">bidirectional version</a>. The only difference is, that the inverse side does not have references to its owners.<br />
<br />
Unidirectional relationship:<br />
<pre class="brush:java">@Entity
public class UnidirectionalManyToOneOwner {
@ManyToOne
private UnidirectionalManyToOneInverse inverse;
}
@Entity
public class UnidirectionalManyToOneInverse {
// nothing special in here
}
</pre><br />
<a href="" name="RelationshipsUnidirectionalOneToMany"></a><h5>Unidirectional One-To-Many</h5>Annotated property in an unidirectional one-to-many relationship can reference any number of entities. It is the owner and may keep them either in a collection or in a map. Even through there is no reference back, each entity on the other side can be owned by only one entity.<br />
<br />
<a href="" name="RelationshipsUnidirectionalOneToManyBasicConfiguration"></a><h6>Basic Configuration</h6>The annotated property must be a <code>Collection</code>, a <code>List</code>, a <code>Set</code> or a <code>Map</code>. No other type is allowed. Mark it with <code>@OneToMany</code> annotation. <br />
<br />
If a <code>List</code>, a <code>Set</code> or a <code>Collection</code> is used, nothing else is needed:<br />
<pre class="brush:java">@Entity
public class OneToManyOwner {
@OneToMany
private Collection<onetomanyinverse> inverses;
}
@Entity
public class OneToManyInverse {
// nothing special in here
}
</pre><br />
If a <code>Map</code> is used, the other side must have a property which contains a map key. Place also <code>@MapKey</code> annotation on the map. Annotation attribute <code>name</code> must contain the name of the property with a key:<br />
<pre class="brush:java">@Entity
public class MapOneToManyOwner {
@OneToMany
@MapKey(name="mapKey")
private Map<String, MapOneToManyInverse> owners;
}
@Entity
public class MapOneToManyInverse {
private String mapKey;
}
</pre><br />
The property specified in <code>@MapKey</code> will be used as a key to the map, the same way as in <a href="#RelationshipsBidirectionalOneToManyManyToOneBasicConfiguration">bidirectional version</a>:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void basicMap() {
EntityManager em = factory.createEntityManager();
MapOneToManyOwner owner = em.find(MapOneToManyOwner.class, 1);
MapOneToManyInverse inverse = em.find(MapOneToManyInverse.class, 5);
// mapKey property is a key to the map
Map<String, MapOneToManyInverse> inverses = owner.getInverses();
assertEquals(inverse, inverses.get(inverse.getMapKey()));
em.close();
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsUnidirectionalOneToManyDefaultDatabaseStructure"></a><h6>Default Database Structure</h6>Unless configured otherwise, JPA assumes that the relationship is kept in a table named <code>OwnerTableName_InverseTableName</code>. <br />
<br />
The table should have two foreign key columns: <br />
<ul><li>The column <code>OwnerPropertyName_inversePrimaryKeyColumnName</code> contains ids of inverse entities. It should have foreign key to the inverse table and have <code>UNIQUE</code> constraint on it.</li>
<li>The column <code>OwnerClassName_ownerPrimaryKeyColumnName</code> contains owner id and should have foreign key to the owners table.</li>
</ul><br />
If a <code>Map</code> is used, the inverse table must contain also a column corresponding to the map key property. The map key property should be unique.<br />
<br />
Db structure for a version with a collection: </div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:sql">OneToManyOwner (
id INT NOT NULL,
CONSTRAINT PK_ONETOMANYOWNER PRIMARY KEY (id)
);
CREATE TABLE OneToManyInverse (
id INT NOT NULL,
CONSTRAINT PK_ONETOMANYINVERSE PRIMARY KEY (id)
);
CREATE TABLE OneToManyOwner_OneToManyInverse (
inverses_id INT NOT NULL,
OneToManyOwner_id INT NOT NULL,
UNIQUE (inverses_id)
);
ALTER TABLE OneToManyOwner_OneToManyInverse
ADD CONSTRAINT fk_OneToManyInverse
FOREIGN KEY (inverses_id) REFERENCES OneToManyInverse (id);
ALTER TABLE OneToManyOwner_OneToManyInverse
ADD CONSTRAINT fk_OneToManyOwner
FOREIGN KEY (OneToManyOwner_id) REFERENCES OneToManyOwner (id);
</pre><br />
</div></div><div style="text-align: justify;">Db structure for a version with a map: </div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:sql">CREATE TABLE MapOneToManyOwner (
id INT NOT NULL,
CONSTRAINT PK_MAPONETOMANYOWNER PRIMARY KEY (id)
);
CREATE TABLE MapOneToManyInverse (
id INT NOT NULL,
mapkey VARCHAR(1500) NOT NULL,
CONSTRAINT PK_MAPONETOMANYINVERSE PRIMARY KEY (id),
UNIQUE (mapkey)
);
CREATE TABLE MapOneToManyOwner_MapOneToManyInverse (
MapOneToManyOwner_id INT NOT NULL,
inverses_id INT NOT NULL,
UNIQUE (inverses_id)
);
ALTER TABLE MapOneToManyOwner_MapOneToManyInverse
ADD CONSTRAINT fk_MapOneToManyInverse
FOREIGN KEY (inverses_id) REFERENCES MapOneToManyInverse (id);
ALTER TABLE MapOneToManyOwner_MapOneToManyInverse
ADD CONSTRAINT fk_MapOneToManyOwner
FOREIGN KEY (MapOneToManyOwner_id) REFERENCES MapOneToManyOwner (id);
</pre><br />
</div></div><div style="text-align: justify;">Note: latest OpenJPA implementation (2.1.1) has a bug:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext">The name of the column referencing the inverse <a href="https://issues.apache.org/jira/browse/OPENJPA-2126">is wrong</a>. OpenJPA uses <code>element_id</code> as a default column name for the this column.</div></div><a href="" name="RelationshipsUnidirectionalOneToManyLoading"></a><h6>Loading</h6>Unidirectional one-to-many relationship is lazy loading by default.<br />
<br />
<a href="" name="RelationshipsUnidirectionalOneToManyCascading"></a><h6>Cascading</h6>All cascading types are compatible with unidirectional one-to-many relationship.<br />
<br />
<a href="" name="RelationshipsUnidirectionalOneToManyConsistency"></a><h6>Consistency</h6>Each inverse can be referenced by only one owner entity. You are responsible for keeping it this way. The safe way how to do it, is to set the inverse column in the joining table as <code>UNIQUE</code>. JPA will throw an exception if somebody tries to save incorrect data.<br />
<br />
Additionally, if you used a map, be careful about changing the value of the map key property. As the inverse has no reference to the owner, it is not possible to write safe setter. Be aware of the issue.<br />
<br />
<a href="" name="RelationshipsUnidirectionalOneToManyOrphanRemoval"></a><h6>Orphan Removal</h6>An orphan in an unidirectional one-to-many relationship is an inverse entity that has no owner. It is the same thing as the orphan in the <a href="#RelationshipsBidirectionalOneToManyManyToOneOrphanRemoval">bidirectional version</a>, only the sides changed.<br />
<br />
If the <code>orphanRemoval</code> attribute of <code>@OneToMany</code> annotation is set to true, JPA will automatically delete orphans. In other words, if you remove the entity from the annotated collection or map, JPA will delete it from the database.<br />
<br />
Set orphan removal:</div><pre class="brush:java">@OneToMany(orphanRemoval=true)
private Collection<OrphanOneToManyInverse> inverses;
</pre><div style="text-align: justify;"><br />
Delete all inverses referenced by the owner:</div><pre class="brush:java">// load the owner and create orphan
OrphanOneToManyOwner owner = em.find(OrphanOneToManyOwner.class, 1);
owner.getInverses().clear();
// commit the transaction - referenced entities are removed
em.getTransaction().commit();
</pre><div style="text-align: justify;"><br />
Expand to see full orphan removal test:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void orphanRemovalOneToMany() {
// check initial data - inverse is available
assertEntityExists(OrphanOneToManyInverse.class, 5);
EntityManager em = factory.createEntityManager();
// load the owner and inverse
OrphanOneToManyOwner owner;
owner = em.find(OrphanOneToManyOwner.class, 1);
OrphanOneToManyInverse inverse;
inverse = em.find(OrphanOneToManyInverse.class, 5);
// remove orphan
em.getTransaction().begin();
owner.getInverses().remove(inverse);
// commit the transaction and close manager
em.getTransaction().commit();
em.close();
// inverse without owner has been removed
assertEntityNOTExists(OrphanOneToManyInverse.class, 5);
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsUnidirectionalOneToManyCustomJoinColumn"></a><h6>Custom Join Column</h6>The <code>@JoinColumn</code> annotation is not compatible with the unidirectional one-to-many relationship. It may seem to work in some contexts, but it fails in others. <br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;">We used the <code>@JoinColumn</code> annotation to annotate the relationship between the <code>ColumnOneToManyOwner</code> and the <code>ColumnOneToManyInverse</code> tables. Following test shows situation, where such configuration does not work. The test is available in the <code>OneToMany_ManyToOneTestCase</code> test case:<br />
<pre class="brush:java">@Test
public void insertUpdateLoadColumnOneToManyOwner() {
//WARNING: this test may behave unpredictably
ColumnOneToManyOwner owner = new ColumnOneToManyOwner(10);
insert(owner);
ColumnOneToManyInverse inverse = new ColumnOneToManyInverse(9);
insert(inverse);
//reload owner and inverse with another entity manager
EntityManager em = factory.createEntityManager();
inverse = em.find(ColumnOneToManyInverse.class, 9);
owner = em.find(ColumnOneToManyOwner.class, 10);
em.close();
//create reference between owner and inverse and save them
owner.setInverses(Arrays.asList(inverse));
mergeAndCommit(inverse);
mergeAndCommit(owner);
//load the owner again
em = factory.createEntityManager();
owner = em.find(ColumnOneToManyOwner.class, 10);
//if the reference would be saved, the owner would have 1 inverse
assertEquals(0, owner.getInverses().size());
em.close();
}
</pre><br />
</div></div>If you wish to customize joining table columns, use the <a href="#RelationshipsUnidirectionalOneToManyCustomJoinTable"><code>@JoinTable</code> annotation</a>. <br />
<br />
<a href="" name="RelationshipsUnidirectionalOneToManyCustomJoinTable"></a><h6>Custom Join Table</h6>The joining table can be customized with <code>@JoinTable</code> annotation. You can customize three attributes: <br />
<ul><li><code>name</code> - table name,</li>
<li><code>joinColumns</code> - column containing owner id,</li>
<li><code>inverseJoinColumns</code> - column containing inverse id.</li>
</ul><br />
If you leave some attribute unspecified, JPA will assume the default name. <br />
<br />
Example configuration:</div><pre class="brush:java">@Entity
public class TableMtmOwner {
private String name;
@ManyToMany
@JoinTable(name="TableMtmJoin",
joinColumns=@JoinColumn(name="owner"),
inverseJoinColumns=@JoinColumn(name="inverse")
)
private Collection<TableMtmInverse> inverses;
}
@Entity
public class TableMtmInverse {
@ManyToMany(mappedBy="inverses")
@MapKey(name="name")
private Map<String, TableMtmOwner> owners;
}
</pre><div style="text-align: justify;"><br />
This works even if you have customized entity id columns or entity table names. The <code>referencedColumnName</code> and <code>table</code> attributes are not necessary:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to expand the test case:</a><br />
<div class="answertext" style="text-align: justify;"><br />
<pre class="brush:java">@Entity
public class TableOneToManyOwner {
@Id
@Column(name="owner_id")
private long id;
@OneToMany
@JoinTable(name="OneToManyJoinTable",
joinColumns=@JoinColumn(name="owner"),
inverseJoinColumns=@JoinColumn(name="inverse")
)
private Collection<TableOneToManyInverse> inverses;
}
@Entity
public class TableOneToManyInverse {
@Id
@Column(name="inverse_id")
private long id;
}
</pre><br />
</div></div><a href="" name="RelationshipsManyToMany"></a><h5>Many-To-Many</h5>Two entity classes have many-to-many relationship if each side is able to reference any number of entities and keeps them in a collection or in a map.<br />
<br />
For example, twitter accounts and their followers have many-to-many relationship. Each account can be followed by any number of people and each person can follow any number of twitter accounts.<br />
<br />
<a href="" name="RelationshipsManyToManyBasicConfiguration"></a><h6>Basic Configuration</h6>Place the <code>@ManyToMany</code> annotation on both sides of the relationship. The annotated property must be a <code>Collection</code>, a <code>List</code>, a <code>Set</code> or a <code>Map</code>. No other type is allowed. <br />
<br />
Decide which side is going to be the owner and which side will be the inverse. You may pick sides at random, many-to-many relationship is symmetric.<br />
<br />
The inverse side must specify the <code>mappedBy</code> annotation attribute. If it is missing, JPA treats the relationship as two separate unidirectional relationships. The attribute contains the name of the owners property that keeps the relationship. The owner side can not have the <code>mappedBy</code> attribute specified. <br />
<br />
If the property is either a <code>List</code>, a <code>Set</code> or a <code>Collection</code>, nothing else is needed. If the property is a <code>Map</code>, you have to use the <code>@MapKey</code> annotation to specify the map key property:</div><pre class="brush:java">@Entity
public class MtmOwner{
private String name;
@ManyToMany
private Collection<MtmInverse> inverses = new ArrayList<MtmInverse>();
}
@Entity
public class MtmInverse{
@ManyToMany(mappedBy="inverses")
@MapKey(name="name")
private Map<String, MtmOwner> owners = new HashMap<String, MtmOwner>();
}
</pre><div style="text-align: justify;"><br />
The property specified in <code>@MapKey</code> will be used as a key to the map, the same way as in <a href="#RelationshipsBidirectionalOneToManyManyToOneBasicConfiguration">bidirectional one-to-many version</a>:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:java">@Test
public void basicMap() {
EntityManager em = factory.createEntityManager();
MtmOwner owner = em.find(MtmOwner.class, 1);
MtmInverse inverse = em.find(MtmInverse.class, 5);
// mapKey property is a key to the map
Map<String, MtmOwner> owners = inverse.getOwners();
assertEquals(owner, owners.get(owner.getName()));
em.close();
}
</pre><br />
</div></div><div style="text-align: justify;"><a href="" name="RelationshipsManyToManyDefaultDatabaseStructure"></a><h6>Default Database Structure</h6>Unless configured otherwise, JPA assumes that the relationship is kept in a table named <code>OwnerTableName_InverseTableName</code>. <br />
<br />
The table should have two foreign key columns: <br />
<ul><li>the column <code>OwnerPropertyName_inversePrimaryKeyColumnName</code> contains ids of inverse entities and should have foreign key to the inverse table,</li>
<li>the column <code>InversePropertyName_ownerPrimaryKeyColumnName</code> contains owner id and should have foreign key to the owners table.</li>
</ul><br />
If a <code>Map</code> is used, the owner the corresponding map key property should be unique.<br />
<br />
The previous relationship corresponds to following db structure:</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><br />
<pre class="brush:sql">CREATE TABLE MtmOwner (
id INT NOT NULL, name VARCHAR(1500),
name VARCHAR(1500) NOT NULL,
CONSTRAINT PK_MTMOWNER PRIMARY KEY (id),
UNIQUE (name)
);
CREATE TABLE MtmInverse (
id INT NOT NULL,
CONSTRAINT PK_MTMINVERSE PRIMARY KEY (id)
);
CREATE TABLE MtmOwner_MtmInverse (
inverses_id INT NOT NULL,
owners_id INT NOT NULL
);
ALTER TABLE MtmOwner_MtmInverse
ADD CONSTRAINT fk_MtmInverse
FOREIGN KEY (inverses_id) REFERENCES MtmInverse (id);
ALTER TABLE MtmOwner_MtmInverse
ADD CONSTRAINT fk_MtmOwner
FOREIGN KEY (owners_id) REFERENCES MtmOwner (id);
</pre><br />
</div></div><div style="text-align: justify;">Note: latest OpenJPA implementation (2.1.1) has a bug:<br />
<div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext">The name of the column referencing the owner <a href="https://issues.apache.org/jira/browse/OPENJPA-2127">is wrong</a>. OpenJPA uses <code>OwnerEntityName_ownerPrimaryKeyColumnName</code> as a default column name for the this column.</div></div><a href="" name="RelationshipsManyToManyCascading"></a><h6>Cascading</h6>The <code>REMOVE</code> cascading type is not compatible with many-to-many relationship. It supports only <code>PERSIST</code>, <code>MERGE</code>, <code>REFRESH</code> and <code>DETACH</code>.<br />
<br />
<a href="" name="RelationshipsManyToManyLoading"></a><h6>Loading</h6>Many-to-many relationship is lazy loading by default.<br />
<br />
<a href="" name="RelationshipsManyToManyConsistency"></a><h6>Consistency</h6>If the relationship is bidirectional, you are responsible for keeping it consistent. The safe way how to do it, is to hide the collection or map and expose only safe consistency keeping 'add', 'remove', ... methods.</div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><div style="text-align: justify;">One possible way how to write safe accessor methods is available in <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/tree/master/src/main/java/org/meri/jpa/relationships/entities/bestpractice"><code>org.meri.jpa.relationships.entities.bestpractice</code></a> package in our demo project. Related test case is named <code>SafeRelationshipsTestCase</code>.</div><pre class="brush:java">/**
* Returns a collection with followed twitter accounts. The
* returned collection is a defensive copy.
*
* @return a collection with followed twitter accounts
*/
public Map<String, SafeTwitterAccount> getTwitterFollowing() {
//defensive copy, nobody will be able to change the
//list from the outside
return new HashMap<String, SafeTwitterAccount>(twitterFollowing);
}
/**
* Add new account to the list of followed twitter accounts.
* The method keeps relationships consistency:
* * this person is set as the account follower also at the twitter side
*/
public void startFollowingTwitter(SafeTwitterAccount account) {
//prevent endless loop
if (twitterFollowing.containsValue(account))
return ;
//add new account
twitterFollowing.put(account.getAccountName(), account);
//set myself into the twitter account
account.addFollower(this);
}
/**
* Removes the account from the list of followed twitter
* accounts. The method keeps relationships consistency:
* * this person is removed from the account followers
* also at the twitter side
*/
public void stopFollowingTwitter(SafeTwitterAccount account) {
//prevent endless loop
if (!twitterFollowing.containsValue(account))
return ;
//remove the account
twitterFollowing.remove(account.getAccountName());
//remove myself from the twitter account
account.removeFollower(this);
}
</pre><br />
</div></div><div style="text-align: justify;">Additionally, if you used a map, be careful about changing the value of the map key property. </div><div class="answer"><a class="answertext" href="javascript:void()" onclick="toggleAnswer(this)">Click to collapse</a><br />
<div class="answertext"><div style="text-align: justify;">One possible way how to write safe accessor methods is available in <code>org.meri.jpa.relationships.entities.bestpractice</code> package <a href="https://github.com/SomMeri/org.meri.jpa.tutorial/tree/master/src/main/java/org/meri/jpa/relationships/entities/bestpractice">on Github</a>. Related test case is named <code>SafeRelationshipsTestCase</code>.</div><pre class="brush:java">public void setAccountName(String accountName) {
Set<SafePerson> allFollowers =
new HashSet<SafePerson>(followers);
for (SafePerson follower : allFollowers) {
follower.stopFollowingTwitter(this);
}
this.accountName = accountName;
for (SafePerson follower : allFollowers) {
follower.startFollowingTwitter(this);
}
}
</pre><br />
This method has one drawback: depending on the JPA implementation, it may cause unnecessary database calls.</div></div><div style="text-align: justify;"><a href="" name="RelationshipsManyToManyOrdering"></a><h6>Ordering</h6>The ordering works exactly the same way as in <a href="#RelationshipsBidirectionalOneToManyManyToOneOrdering">one-to-many relationship</a>.<br />
<br />
If the relationship uses a list, it may be sorted with the <code>@OrderBy</code> annotation. Put property name followed by <code>ASC</code> or <code>DESC</code> into its value and JPA will use it to sort loaded entities. If you need multiple sorting properties, separate them with comma ','.<br />
<br />
Sort by the <code>ordering</code> property. Entities with equal <code>ordering</code> property are sorted by <code>id</code>:</div><pre class="brush:java">@Entity
public class OrderedMtmOwner {
@ManyToMany
@OrderBy("ordering ASC, id DESC")
private List<OrderedMtmInverse> inverses =
new ArrayList<OrderedMtmInverse>();
}
@Entity
public class OrderedMtmInverse {
@Id
private long id;
private String ordering;
@ManyToMany(mappedBy="inverses")
@MapKey(name="name")
private Mapt<String, OrderedMtmOwnert> owners =
new HashMapt<String, OrderedMtmOwner>();
}
</pre><div style="text-align: justify;"><br />
<a href="" name="RelationshipsManyToManyCustomJoinColumn"></a><h6>Custom Join Column</h6>The <code>@JoinColumn</code> annotation is not compatible with the many-to-many relationship. If you wish to customize joining table columns, use the <a href="#RelationshipsManyToManyCustomJoinTable"><code>@JoinTable</code> annotation</a>. <br />
<br />
<a href="" name="RelationshipsManyToManyCustomJoinTable"></a><h6>Custom Join Table</h6>The joining table can be customized with <code>@JoinTable</code> annotation, exactly the same way as in <a href="#RelationshipsUnidirectionalOneToManyCustomJoinTable">unidirectional one-to-many relationship</a>. You can customize three attributes: <br />
<ul><li><code>name</code> - table name,</li>
<li><code>joinColumns</code> - column containing owner id,</li>
<li><code>inverseJoinColumns</code> - column containing inverse id.</li>
</ul><br />
If you specify only the <code>name</code> attribute, JPA will assume that joining columns have default names. <br />
<br />
Example configuration:<br />
<pre class="brush:java">@Entity
public class TableOneToManyOwner {
@OneToMany
@JoinTable(name="OneToManyJoinTable",
joinColumns=@JoinColumn(name="owner"),
inverseJoinColumns=@JoinColumn(name="inverse")
)
private Collection<TableOneToManyInverse> inverses;
}
@Entity
public class TableOneToManyInverse {
// nothing special here
}
</pre><br />
This works even if you have customized entity id columns or entity table names. The <code>referencedColumnName</code> and <code>table</code> attributes are not necessary.<br />
<br />
<a href="" name="End"></a><h4>End</h4>This post covered only the most basic JPA features. Of course, JPA has much more worth learning about. <br />
<br />
The list of features missing from this post includes: <br />
<ul><li>JPQL Query Language,</li>
<li>class inheritance,</li>
<li>automatic entity id generation,</li>
<li>custom data types,</li>
<li>entities composed of multiple tables,</li>
<li>metadata,</li>
<li>configuration via xml instead of annotations,</li>
<li>entity lifecycle callbacks.</li>
</ul><br />
Some of them are more useful than others. While JPQL and class inheritance are a must, metadata and entity lifecycle callbacks are useful only for some projects. The rest is somewhere in between.</div><script type="text/javascript">
function nextDiv(element) {
do {
element = element.nextSibling;
} while (element && element.nodeName != "DIV");
return element;
}
function getElementsByClass(node, searchClass, tag) {
var classElements = new Array();
var els = node.getElementsByTagName(tag); // use "*" for all elements
var elsLen = els.length;
var pattern = new RegExp("\\b"+searchClass+"\\b");
for (i = 0, j = 0; i < elsLen; i++) {
if ( pattern.test(els[i].className) ) {
classElements[j] = els[i];
j++;
}
}
return classElements;
}
function expand(heading, answer) {
answer.style.display="block";
heading.innerHTML="[-] Click to Collapse";
}
function collapse(heading, answer) {
answer.style.display="none";
heading.innerHTML="[+] Click to Expand";
}
function toggleAnswer(heading) {
answer=nextDiv(heading);
if (answer.style.display=="none") {
expand(heading, answer);
} else {
collapse(heading, answer);
}
}
function expandAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++) {
expand(headings[i], nextDiv(headings[i]));
}
}
function collapseAllAnswers() {
var headings = getElementsByClass(document, 'answertext', 'a');
for(i=0; i<headings.length; i++)
collapse(headings[i], nextDiv(headings[i]));
}
window.onload = collapseAllAnswers;
</script>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com131tag:blogger.com,1999:blog-3355333693246614846.post-23146550105826196482012-02-01T00:01:00.000-08:002012-02-01T01:06:18.993-08:00Running JNDI and JPA Without J2EE Container<div style="text-align: justify;">We wanted to test some JPA code with as simple setup as possible. The plan was to use only Java and Maven, without an application server or other J2EE container.<br />
<br />
Our JPA configuration needs two things to run successfully:</div><ul style="text-align: justify;"><li>database to store data,</li>
<li>JNDI to access the database.</li>
</ul><div style="text-align: justify;"><br />
This post has two parts. First part shows how to use standalone JNDI and an embedded in-memory database in test. Remaining chapters explain how the solution works.<a name='more'></a><br />
<br />
All used code is available <a href="https://github.com/SomMeri/Standalone-JPA-Test-Demo">on Github</a>. If you are interested in the solution but do not want to read the explanations, download the project from Github and read only the first chapter.<br />
<br />
</div><h4 style="text-align: justify;">Table of Contents</h4>List of chapters:<br />
<ul><li><a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#JPATest">JPA Test</a></li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#JNDIBasics">JNDI</a></li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#StandaloneJNDI">Standalone JNDI</a></li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#InMemoryDatabase">In-Memory Database</a></li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#DatabaseStructure">Database Structure</a></li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#End">End</a></li>
</ul><br />
<a href="" name="JPATest"></a><br />
<h4 style="text-align: justify;">JPA Test</h4><div style="text-align: justify;">This chapter shows how to use <a href="https://github.com/SomMeri/Standalone-JPA-Test-Demo">our code</a> to enable standalone JNDI and embedded in-memory database in tests. How and why the solution works is explained in the rest of this post.<br />
<br />
The solution has three 'API' classes: <br />
<ul><li><a href="https://github.com/SomMeri/Standalone-JPA-Test-Demo/blob/master/src/test/java/org/meri/jpa/util/JNDIUtil.java#L14"><code>JNDIUtil</code></a> - JNDI initialization, clean up and some convenience methods,</li>
<li><a href="https://github.com/SomMeri/Standalone-JPA-Test-Demo/blob/master/src/test/java/org/meri/jpa/util/InMemoryDBUtil.java#L17"><code>InMemoryDBUtil</code></a> - database and data source creation/removal, </li>
<li><a href="https://github.com/SomMeri/Standalone-JPA-Test-Demo/blob/master/src/test/java/org/meri/jpa/AbstractTestCase.java#L13"><code>AbstractTestCase</code></a> - database clean up before first test and JNDI clean up before each test.</li>
</ul><br />
We use <a href="http://www.liquibase.org/">Liquibase</a> to maintain the database structure. If you do not wish to use Liquibase, you will have to customize the class <code>InMemoryDBUtil</code>. Tweak the method <code>createDatabaseStructure</code> to do what you need.<br />
<br />
Liquibase keeps list of all needed database changes in a file named changelog. Unless configured otherwise, each change runs only once. Even if the changelog file is applied multiple times to the same database.<br />
<br />
<h5>Usage</h5>Any test case extended from the <code>AbstractTestCase</code> will:<br />
<ul><li>drop the database before first test,</li>
<li>install standalone JNDI or delete all data stored in it before each test,</li>
<li>run Liquibase changelog against the database before each test.</li>
</ul><br />
JPA test case must extend <code>AbstractTestCase</code> and override the <code>getInitialChangeLog</code> method. The method should return changelog file location. </div><pre class="brush:java">public class DemoJPATest extends AbstractTestCase {
private static final String CHANGELOG_LOCATION = "src/test/java/org/meri/jpa/simplest/db.changelog.xml";
private static EntityManagerFactory factory;
public DemoJPATest() {
}
@Override
protected String getInitialChangeLog() {
return CHANGELOG_LOCATION;
}
@Test
@SuppressWarnings("unchecked")
public void testJPA() {
EntityManager em = factory.createEntityManager();
Query query = em.createQuery("SELECT x FROM Person x");
List<Person> allUsers = query.getResultList();
em.close();
assertFalse(allUsers.isEmpty());
}
@BeforeClass
public static void createFactory() {
factory = Persistence.createEntityManagerFactory("Simplest");
}
@AfterClass
public static void closeFactory() {
factory.close();
}
}
</pre><div style="text-align: justify;"><br />
Note: it would be cleaner to drop the database before each test. However, drop and recreate db structure is costly operation. It would slow down the test case too much. Doing it only before the class seems to be reasonable compromise.<br />
<br />
While the database is dropped only once, the changelog is run before each test. It may seems like a waste, but this solution has some advantages. First, the <code>getInitialChangeLog</code> method does not have to be static and may be overridden in each test. Second, changes configured to 'runAlways' will run before each test and thus may contain some cheap clean up or other initialization.<br />
<br />
</div><a href="" name="JNDIBasics"></a><br />
<h4 style="text-align: justify;">JNDI</h4><div style="text-align: justify;">This chapter explains what JNDI is, how it is used and how to configure it. If you are not interested in theory, <a href="http://www.blogger.com/post-edit.g?blogID=3355333693246614846&postID=2314655010582619648#StandaloneJNDI">skip</a> to the next chapter. Standalone JNDI is created there.<br />
<br />
<h5>Basic Usage</h5>JNDI allows clients to store and look up data and objects via a name. The data store is accessed through an implementation of the interface <code>Context</code>. <br />
<br />
The following code shows how to store data in JNDI:<br />
<pre class="brush:java">Context ctx = new InitialContext();
ctx.bind("jndiName", "value");
ctx.close();
</pre><br />
Second piece of code shows how to look for things in JNDI:</div><div style="text-align: justify;"><pre class="brush:java">Context ctx = new InitialContext();
Object result = ctx.lookup("jndiName");
ctx.close();
</pre><br />
</div><div style="text-align: justify;">Try to run the above without a J2EE container and you will get an error:</div><div style="text-align: justify;"><pre class="brush:java">javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial
at javax.naming.spi.NamingManager.getInitialContext(Unknown Source)
at javax.naming.InitialContext.getDefaultInitCtx(Unknown Source)
at javax.naming.InitialContext.getURLOrDefaultInitCtx(Unknown Source)
at javax.naming.InitialContext.bind(Unknown Source)
at org.meri.jpa.JNDITestCase.test(JNDITestCase.java:16)
at ...
</pre></div><div style="text-align: justify;"><br />
The code does not work because <code>InitialContext</code> class is not a real data store. The <code>InitialContext</code> class is only able to find another instance of the <code>Context</code> interface and delegates all work to it. It is not able to store data nor to find them.<br />
<br />
<h5>Context Factories</h5>The real context, the one that does all the work and is able to store/find data, has to be created by a context factory. This section shows how to create a context factory and how to configure <code>InitialContext</code> to use it. <br />
<br />
Each context factory must implement <code>InitialContextFactory</code> interface and must have a no-argument constructor:</div><pre class="brush:java">package org.meri.jpa.jndi;
public class MyContextFactory implements InitialContextFactory {
@Override
public Context getInitialContext(Hashtable environment) throws NamingException {
return new MyContext();
}
}
</pre><div style="text-align: justify;"><br />
Our factory returns a simple context called <code>MyContext</code>. Its <code>lookup</code> method always returns a string "stored value":</div><pre class="brush:java">class MyContext implements Context {
@Override
public Object lookup(Name name) throws NamingException {
return "stored value";
}
@Override
public Object lookup(String name) throws NamingException {
return "stored value";
}
.. the rest ...
}
</pre><div style="text-align: justify;"><br />
JNDI configuration is passed between classes in a hash table. The key always contains property name and the value contains the property value. As the initial context constructor <code>InitialContext()</code> has no parameter, an empty hash table is assumed. The class has also an alternative constructor which takes configuration properties hash table as a parameter. <br />
<br />
Use the property <code>"java.naming.factory.initial"</code> to specify context factory class name. The property is defined in <code>Context.INITIAL_CONTEXT_FACTORY</code> constant.</div><pre class="brush:java">Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, "className");
Context ctx = new InitialContext(environnement);
</pre><div style="text-align: justify;"><br />
Next test configures <code>MyContextFactory</code> and checks whether created initial context returns "stored value" no matter what:</div><pre class="brush:java">@Test
@SuppressWarnings({ "unchecked", "rawtypes" })
public void testDummyContext() throws NamingException {
Hashtable environnement = new Hashtable();
environnement.put(Context.INITIAL_CONTEXT_FACTORY, "org.meri.jpa.jndi.MyContextFactory");
Context ctx = new InitialContext(environnement);
Object value = ctx.lookup("jndiName");
ctx.close();
assertEquals("stored value", value);
}
</pre><div style="text-align: justify;"><br />
Of course, this works only if you can supply hash table with custom properties to the initial context constructor. That is often impossible. Most libraries use no-argument constructor shown in the beginning. They assume that initial context class has default context factory available and that no-argument constructor will use that one.<br />
<br />
<h5>Naming Manager</h5>Initial context uses <code>NamingManager</code> to create a real context. Naming manager has a static method <code>getInitialContext(Hashtable env) </code> which returns an instance of a context. The parameter <code>env</code> contains a configuration properties used to build the context.<br />
<br />
By default, naming manager reads <code>Context.INITIAL_CONTEXT_FACTORY</code> from the <code>env</code> hash table and creates an instance of specified initial context factory. The factory method then creates a new context instance. If that property is not set, naming manager throws an exception.<br />
<br />
It is possible to customize naming managers behavior. The class <code>NamingManager</code> has method <code>setInitialContextFactoryBuilder</code>. If the initial context factory builder is set, naming manager will use it to create context factories.<br />
<br />
You can use this method only once. Installed context factory builder can not be changed. </div><pre class="brush:java">try {
MyContextFactoryBuilder builder = new MyContextFactoryBuilder();
NamingManager.setInitialContextFactoryBuilder(builder);
} catch (NamingException e) {
// handle exception
}
</pre><div style="text-align: justify;"><br />
Initial context factory builder must implement <code>InitialContextFactoryBuilder</code> interface. The interface is simple. It has only one method <code>InitialContextFactory createInitialContextFactory(Hashtable env)</code>.<br />
<br />
<h5>Summary</h5>In short, initial context delegates a real context initialization to naming manager which delegates it to context factory. Context factory is created by an instance of initial context factory builder.<br />
<br />
</div><a href="" name="StandaloneJNDI"></a><br />
<h4 style="text-align: justify;">Standalone JNDI</h4><div style="text-align: justify;">We will create and install standalone JNDI implementation. The entry point to our standalone JNDI implementation is the class <code>JNDIUtil</code>.<br />
<br />
Three things are needed to enable JNDI without an application server:<br />
<ul><li>an implementation of <code>Context</code> and <code>InitialContextFactory</code> interfaces,</li>
<li>an implementation of <code>InitialContextFactoryBuilder</code> interface,</li>
<li>initial context factory builder installation and ability to clean all stored data.</li>
</ul><br />
</div><h5>Context and Factory</h5><div style="text-align: justify;">We took <a href="http://code.google.com/p/osjava/wiki/SimpleJNDI">SimpleJNDI</a> implementation from <a href="http://code.google.com/p/osjava/">osjava</a> project and modified it to suit our needs better. The project uses a <a href="http://www.opensource.org/licenses/bsd-license.php">new BSD license</a>. <br />
<br />
Add SimpleJNDI maven dependency into pom.xml:</div><pre class="brush:xml"><dependency>
<groupid>simple-jndi</groupid>
<artifactid>simple-jndi</artifactid>
<version>0.11.4.1</version>
</dependency>
</pre><div style="text-align: justify;"><br />
SimpleJNDI comes with a <code>MemoryContext</code> context which lives exclusively in the memory. It requires almost no configuration and its state is never saved of loaded. It does almost what we need, except two things:<br />
<ul><li>its <code>close()</code> method deletes all stored data,</li>
<li>each instance uses its own storage by default.</li>
</ul><br />
Most libraries assume that the close method optimizes resources. They tend to call it each time they load or store data. If the close method deletes all data right after they have been stored, the context is useless. We have to extend the <code>MemoryContext</code> class and override the <code>close</code> method:</div><pre class="brush:java">@SuppressWarnings({"rawtypes"})
public class CloseSafeMemoryContext extends MemoryContext {
public CloseSafeMemoryContext(Hashtable env) {
super(env);
}
@Override
public void close() throws NamingException {
// Original context lost all data on close();
// That made it unusable for my tests.
}
}
</pre><div style="text-align: justify;"><br />
By convention, the builder/factory system creates new context instance for each use. If they do not share data, JNDI can not be used to transfer data between different libraries.<br />
<br />
Fortunately, this problem has also an easy solution. If the environnement hash table contains property <code>"org.osjava.sj.jndi.shared"</code> with value <code>"true"</code>, created memory context will use common static storage. Therefore, our initial context factory creates <code>CloseSafeMemoryContext</code> instances and configures them to use common storage:</div><pre class="brush:java">public class CloseSafeMemoryContextFactory implements InitialContextFactory {
private static final String SHARE_DATA_PROPERTY = "org.osjava.sj.jndi.shared";
public Context getInitialContext(Hashtable environment) throws NamingException {
// clone the environnement
Hashtable sharingEnv = (Hashtable) environment.clone();
// all instances will share stored data
if (!sharingEnv.containsKey(SHARE_DATA_PROPERTY)) {
sharingEnv.put(SHARE_DATA_PROPERTY, "true");
}
return new CloseSafeMemoryContext(sharingEnv);;
}
}
</pre><br />
<h5>Initial Context Factory Builder</h5><div style="text-align: justify;">Our builder acts almost the same way as the original naming manager implementation. If the property <code>Context.INITIAL_CONTEXT_FACTORY</code> is present in the incoming environment, specified factory is created. <br />
<br />
However, if this property is missing, the builder creates an instance of <code>CloseSafeMemoryContextFactory</code>. The original naming manager would throw an exception.<br />
<br />
Our implementation of the <code>InitialContextFactoryBuilder</code> interface:</div><pre class="brush:java">public InitialContextFactory createInitialContextFactory(Hashtable env) throws NamingException {
String requestedFactory = null;
if (env!=null) {
requestedFactory = (String) env.get(Context.INITIAL_CONTEXT_FACTORY);
}
if (requestedFactory != null) {
return simulateBuilderlessNamingManager(requestedFactory);
}
return new CloseSafeMemoryContextFactory();
}
</pre><div style="text-align: justify;"><br />
The method <code>simulateBuilderlessNamingManager</code> uses class loader to load requested context factory:</div><pre class="brush:java">private InitialContextFactory simulateBuilderlessNamingManager(String requestedFactory) throws NoInitialContextException {
try {
ClassLoader cl = getContextClassLoader();
Class requestedClass = Class.forName(className, true, cl);
return (InitialContextFactory) requestedClass.newInstance();
} catch (Exception e) {
NoInitialContextException ne = new NoInitialContextException(...);
ne.setRootCause(e);
throw ne;
}
}
private ClassLoader getContextClassLoader() {
return (ClassLoader) AccessController.doPrivileged(new PrivilegedAction() {
public Object run() {
return Thread.currentThread().getContextClassLoader();
}
});
}
</pre><div style="text-align: justify;"><br />
</div><h5>Builder Installation and Context Cleaning</h5><div style="text-align: justify;">Finally, we have to install context factory builder. As we wanted to use standalone JNDI in tests, we needed also a method to clean up all stored data between tests. Both is done inside the method <code>initializeJNDI</code> which will run before each test:</div><pre class="brush:java">public class JNDIUtil {
public void initializeJNDI() {
if (jndiInitialized()) {
cleanAllInMemoryData();
} else {
installDefaultContextFactoryBuilder();
}
}
}
</pre><div style="text-align: justify;"><br />
JNDI is initialized if the default context factory builder has been set already:</div><pre class="brush:java">private boolean jndiInitialized() {
return NamingManager.hasInitialContextFactoryBuilder();
}
</pre><div style="text-align: justify;"><br />
Installation of the default context factory builder:</div><pre class="brush:java">private void installDefaultContextFactoryBuilder() {
try {
NamingManager.setInitialContextFactoryBuilder(new ImMemoryDefaultContextFactoryBuilder());
} catch (NamingException e) {
//We can not solve the problem. We will let it go up without
//having to declare the exception every time.
throw new ConfigurationException(e);
}
}
</pre><div style="text-align: justify;"><br />
Use the original implementation of the method <code>close</code> in <code>MemoryContext</code> class to clean up stored data:</div><pre class="brush:java">private void cleanAllInMemoryData() {
CleanerContext cleaner = new CleanerContext();
try {
cleaner.close();
} catch (NamingException e) {
throw new RuntimeException("Memory context cleaning failed:", e);
}
}
class CleanerContext extends MemoryContext {
private static Hashtable environnement = new Hashtable();
static {
environnement.put("org.osjava.sj.jndi.shared", "true");
}
public CleanerContext() {
super(environnement);
}
}
</pre><div style="text-align: justify;"><br />
</div><a href="" name="InMemoryDatabase"></a><br />
<h4 style="text-align: justify;">In-Memory Database</h4><div style="text-align: justify;"><a href="http://db.apache.org/derby/">Apache Derby</a> is an open source relational database implemented in Java. It is available under Apache License, Version 2.0. Derby is able to run in embedded mode. Embedded database data are stored either on the filesystem or in the memory. <br />
<br />
Maven dependency for Derby:<br />
<pre class="brush:xml"><dependency>
<groupid>org.apache.derby</groupid>
<artifactid>derby</artifactid>
<version>10.8.2.2</version>
</dependency>
</pre><br />
<h5 style="text-align: justify;">Create DataSource</h5>Use an instance of the <code>EmbeddedDatasource</code> class to connect to the database. The data source will use an in-memory instance whenever the database name starts with "memory:". <br />
<br />
Following code creates data source pointing to an instance of in-memory database. If the database does not exist yet, it will be created:<br />
<pre class="brush:java">private EmbeddedDataSource createDataSource() {
EmbeddedDataSource dataSource = new EmbeddedDataSource();
dataSource.setDataSourceName(dataSourceJndiName);
dataSource.setDatabaseName("memory:" + databaseName);
dataSource.setCreateDatabase("create");
return dataSource;
}
</pre><br />
<h5 style="text-align: justify;">Drop Database</h5>The easiest way to clean up the database is to drop and recreate it. Create an instance of embedded data source, set the connection attribute "drop" to "true" and call its <code>getConnection</code> method. It will drop the database and throw an exception. <br />
<pre class="brush:java">private static final String DATABASE_NOT_FOUND = "XJ004";
private void dropDatabase() {
EmbeddedDataSource dataSource = createDataSource();
dataSource.setCreateDatabase(null);
dataSource.setConnectionAttributes("drop=true");
try {
//drop the database; not the nicest solution, but works
dataSource.getConnection();
} catch (SQLNonTransientConnectionException e) {
//this is OK, database was dropped
} catch (SQLException e) {
if (DATABASE_NOT_FOUND.equals(e.getSQLState())) {
//attempt to drop non-existend database
//we will ignore this error
return ;
}
throw new ConfigurationException("Could not drop database.", e);
}
}
</pre><br />
<a href="" name="DatabaseStructure"></a><br />
<h4 style="text-align: justify;">Database Structure</h4>We used <a href="http://www.liquibase.org/">Liquibase</a> to create the database structure and test data. The database structure is kept in a so called changelog file. It is an xml file, but you can include DDL or SQL code if you do not feel like learning yet another xml language. <br />
<br />
Liquibase and its advantages are out of scope of this article. The most relevant advantage for this demo is its ability to run the same changelog against the same database multiple times. Each run applies only new changes to the database. If the file did not changed, nothing happens.<br />
<br />
You can add the changelog to the jar or war and run it on each application start up. That will ensure that the database is always updated to the latest version. No configuration or installation scripts are necessary.<br />
<br />
Add Liquibase dependency to the pom.xml:<br />
<pre class="brush:xml"><dependency>
<groupid>org.liquibase</groupid>
<artifactid>liquibase-core</artifactid>
<version>2.0.3</version>
</dependency>
</pre><br />
Following changelog creates one table named Person and puts one entry 'slash - Simon Worth' into it:</div><pre class="brush:xml"><databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">
<changeSet id="1" author="meri">
<comment>Create table structure for users and shared items.</comment>
<createTable tableName="person">
<column name="user_id" type="integer">
<constraints primaryKey="true" nullable="false" />
</column>
<column name="username" type="varchar(1500)">
<constraints unique="true" nullable="false" />
</column>
<column name="firstname" type="varchar(1500)"/>
<column name="lastname" type="varchar(1500)"/>
<column name="homepage" type="varchar(1500)"/>
<column name="about" type="varchar(1500)"/>
</createTable>
</changeSet>
<changeSet id="2" author="meri" context="test">
<comment>Add some test data.</comment>
<insert tableName="person">
<column name="user_id" valueNumeric="1" />
<column name="userName" value="slash" />
<column name="firstName" value="Simon" />
<column name="lastName" value="Worth" />
<column name="homePage" value="http://www.slash.blogs.net" />
<column name="about" value="I like nature and writing my blog. The blog contains my opinions about everything." />
</insert>
</changeSet>
</databaseChangeLog>
</pre><div style="text-align: justify;"><br />
Liquibase use is pretty straightforward. Use data source to create new <code>Liquibase</code> instance, run its <code>update</code> method and handle all declared exceptions:</div><pre class="brush:java">private void initializeDatabase(String changelogPath, DataSource dataSource) {
try {
//create new liquibase instance
Connection sqlConnection = dataSource.getConnection();
DatabaseConnection db = new DerbyConnection(sqlConnection);
Liquibase liquibase = new Liquibase(changelogPath, new FileSystemResourceAccessor(), db);
//update the database
liquibase.update("test");
} catch (SQLException e) {
// We can not solve the problem. We will let it go up without
// having to declare the exception every time.
throw new ConfigurationException(DB_INITIALIZATION_ERROR, e);
} catch (LiquibaseException e) {
// We can not solve the problem. We will let it go up without
// having to declare the exception every time.
throw new ConfigurationException(DB_INITIALIZATION_ERROR, e);
}
}
</pre><div style="text-align: justify;"><br />
<a href="" name="End"></a><br />
<h4 style="text-align: justify;">End</h4>Both standalone JNDI and embedded in-memory database are up and running each time we run our tests. While the JNDI set up is probably universal, the database construction will probably need project specific modifications. <br />
<br />
Feel free to download sample project from <a href="https://github.com/SomMeri/Standalone-JPA-Test-Demo">Github</a> and use/modify whatever you find useful.</div>Merihttp://www.blogger.com/profile/15636826095399788217noreply@blogger.com8