Latest posts for the topic "Lazy loading / Data on demand / Paging big tables solution"

Lazy loading / Data on demand / Paging big tables solution

jonsi — GMT

Hi all, I hope this post will ease people the pain when it comes to working with huge data sets. I've done allot of searching, googling and here I present my findings that will hopefully save you some hours. Credits go to Roger at www.ilikespam.com. I highly recommend that this sort of stuff should be well documented in IceFaces tutorials and developers guides (untill the planed component will come to existence in IceFaces 2.0) The problem: IceFaces objective is to provide enterprise developers some handy tools when creating some web apps (and does a damn good job at it imo). I (and most likely many others) however found one stone left unturned, in enterprise development it is very common that you have to work with queries that return huge data sets. When you use a ice:dataTable and a ice:dataPaginator to presents this data you'll have to provide the ice:dataTable with a list or an array of objects (business logic objects most of the time). As in my case, creating and returning a list with few thousand entries was simply not an option and probably not for anyone (unless you've got 1337 TB worth of RAM). Solution: I came across 2 solutions, one much better than the other. I'll not go into much detail of the first one but here's the link to that forum post (note: I did try this one out but couldn't get it to work for me). The other solution is much more simple and makes perfect sense once you realize it. In essence you simply create a new class (LazyLoading.java for example) and make it extend java.util.AbstractList. What you do then is simply "lie" how big the list is because the ice:dataPaginator uses the size() method in the list to find out the entries count. Then you implement your own get(int index) method which loads data as they become needed (when you flip the page). A link to the page with the second method (the elegant and simple one) can be found here! Exmple of how it works: You've got a page with a and a assigned to it with 10 items per page. Your backingbean has a instance of LazyLoadingList (lazyList) which you initialize with a query object, page size and the number of results the query should have returned (this means in fact that you have to do 2x queries every time, however the count(*) query is usually very lite). In the backingbean a instance of LazyLoadingList is returned to the dataTable. The page loads and the LazyLoadingList makes a query for the first 10 items (IceFaces calls lazyList.get(0), ..., lazyList.get(9) to fill in the table). Then lets say you press any page (other than the first) in the dataPaginator, that triggers IceFaces to refresh the dataTable and to fetch new items (somewhat in the lines of lazyList.get(10) for the first item on the second page). The LazyLoadingList get(int index) method checks if it has the requested indexes in range, if it doesn't (a new page) it does a query for it. My way: Now I did it somewhat different than on the web page I referred to but it was a great starting point for me. What I did different was I made a interface LazyLoadable.java that has the method getUpdatedList(int index, int pageSize) that returns a list and I make my backingbean implement that method. This is the method that actually queries the databae. I do this because I've already got my database related objects initialized there instead of creating them again or passing them as parameters (whatever suits you). In the constructor of LazyLoadingList I take LazyLoadingList(LazyLoadable lazyObject, int pageSize, int numResults). Then instead of a HashTable I used 2x lists. When using this I've a common refresh method in my backingbean (column sort, filter change etc.). So every time something changes this method gets called and in it I recreate the list as follows:

this.businessObjectList = new LazyLoadingList(this, dataTable.getRows(), myQueryObject.count()); dataTable: is a global HtmlDataTable instance so I define how many rows I want in the .jspx document. myQueryObject.count(): returns how many results the query really returns (could be couple of thousand or whatnot). In addition to this I've the usual getBusinessObjectList() method that gets wired something like this (note: just an example, in practise I've a lot more of them): < ice:dataTable rows="10" value="#{backingbean.businessObjectList}" > I'll attach those 2x files as attachment and I belive the code speaks for itself (the comments were in Icelandic so I removed them but I've described everything in English here in this post). Feel free ask around and we'll see if I can be of any help. Cheers from Iceland (kinda appropriate we use IceFaces and we're from Iceland don't you think?) -_- Keywords: lazy loading data on demand paging big tables huge query datapaginator paging

Re:Lazy loading / Data on demand / Paging big tables solution

nerlijma — GMT

Hi, I have tried you example (very nice) and pagination works fine. However, with 10 million of rows and a RowSelector, when selecting an item it takes for ever. If you have a look at RowSelector.java (309) where Code:

// ICE-2024: should clear the whole table, not just the displayed page.
if (!getMultiple().booleanValue()) {
   for (int i = 0; i < dataTable.getRowCount(); i++) {
  if (i != rowIndex) {
  dataTable.setRowIndex(i);
  setValue(Boolean.FALSE);
  }
  }
 }

you can see that ICEFaces is iterating all 10 millon of rows to clean the selected value because is calling getRowCount(). Are you selecting a row in your table and it is selected properly? if not, have you any workaround for this?

Re:Lazy loading / Data on demand / Paging big tables solution

jonsi — GMT

Sup nerlijma. We checked it out and you're right. We are now working on new project where we use a the ice:rowSelector component and uppon inspection we discovered that the RowSlector indeed iterates over the entire list (the LazyLoadingList fetches new rows as they become needed). Not only does it iterate the entire list, it does it quite often (we haven't found a pattern yet but it iterates the list 2 or 3 times). Along with this we did some debugging and also found out that when IceFaces fills in the table it fetches every page 2 times. Atm we're trying to figure it out and will post our findings here. Those two projects we're working on were sortof a tryout for IceFaces, while the concept is nice its those practical kinks that makes us less and less likely to adopt it. cheers, jonsi

Re:Lazy loading / Data on demand / Paging big tables solution

rdefuentes — GMT

HI all, Rowselector is not working for me with this lazy pattern too, as nerlijma posted, it is iterating the whole list. I will post a very simple example to the ICEfaces team to have a look. Huge tables is a quite often requirement !! best regards

Re:Lazy loading / Data on demand / Paging big tables solution

mark.collette — GMT

One work-around is to declare your rowSelector as using multiple selection, and then in your selectionListener, enforce the single selection yourself. This is easiest done when using a smart Map to keep your selections, instead of using a per row field for selection.

Lazy loading / Data on demand / Paging big tables solution

nerlijma — GMT

Mark, I have changed RowSelector.java adding a lastRowIndex used only if is not multiple. Also commented where it iterates over all rows and uncommented the old code, where if is not the selected row, rowselector marks it as FALSE. Therefore, if is not multiple, with every click, old selected index is set to FALSE as well as not clicked ones. Only clicked row index is set to true. I am using 16 Million rows dataTable, with a LazyList pattern, and works perfect. Code:

        if (!getMultiple().booleanValue()) {
                dataTable.setRowIndex(lastClickedRow);
                setValue(Boolean.FALSE);
                lastClickedRow = rowIndex;
        }
                
                // ICE-2024: should clear the whole table, not just the
                // displayed page.
//                if (!getMultiple().booleanValue()) {
// this code all commented
//                }

            } else {
                // Uncommented this...
                if (Boolean.FALSE.equals(rowSelector.getMultiple())) {
                    // Clear all other selections
                    rowSelector.setValue(Boolean.FALSE);
                }
            }

Re:Lazy loading / Data on demand / Paging big tables solution

bchi49 — GMT

Here's a generic version, please review: public class LazyLoadingList extends AbstractList { private LazyLoadable lazyObject; private List firstList; private List viewList; private int currentPage = -1; private int numResults; private int pageSize; public LazyLoadingList(LazyLoadable lazyObject, int pageSize, int numResults) { this.lazyObject = lazyObject; this.numResults = numResults; this.pageSize = pageSize; } @Override public E get(int i) { if (i < pageSize) { if (firstList == null) firstList = lazyObject.getUpdatedList(i, pageSize); return (E)firstList.get(i); } int page = i / pageSize; if (page != currentPage) { currentPage = page; viewList = lazyObject.getUpdatedList(i, pageSize); } return viewList.get(i % pageSize); } @Override public int size() { return numResults; } public void setNumResults(int numResults) { this.numResults = numResults; } } public interface LazyLoadable { public List getUpdatedList(int index, int pageSize); }

Lazy loading / Data on demand / Paging big tables solution

kumareshav — GMT

can i have some examples with this lazyloading class being used with data table?

Re:Lazy loading / Data on demand / Paging big tables solution

trumaxformax1 — GMT

Hi. I need to implement some method to show big tables too. Reading the comments I not sure If you try to implement a paging or something like a live-scroll. I have found a component in http://www.nitobi.com/products/completeui/demos/explorer/ which show what I need. No matter how many records you have it show as you scroll down/up.... My question is: Work you example as this??? Thanks.

Re:Lazy loading / Data on demand / Paging big tables solution

kumareshav — GMT

I will get back to u once i finished look into this site which you have given. Thanks

Re:Lazy loading / Data on demand / Paging big tables solution

trumaxformax1 — GMT

Ok, thanks. I'll be waiting. Greetings

Re:Lazy loading / Data on demand / Paging big tables solution

brad.kroeger — GMT

I have completed a tutorial covering lazy loading a dataTable with JPA and the ICEfaces rendering api. It is located at http://anonsvn.icefaces.org/repo/tutorials/trunk/tutorials/samples/dataTable-JPA.

Re:Lazy loading / Data on demand / Paging big tables solution

brad.kroeger — GMT

Here is the proper link, the docs folder has the html page with the tutorial content. The tutorial should be published to icefaces.org in the near future. http://anonsvn.icefaces.org/repo/tutorials/trunk/tutorials/samples/dataTable-JPA

Lazy loading / Data on demand / Paging big tables solution

ybendek — GMT

Hi all. well I'm trying Jonsi option to paging a lot of records, it works fine...however when I tryied to log what is the real query running with the database... something happen :S. I have an DAO object implementing LazyLoadable interface, checking what is the real request I have some log in it to catch every SQL request.. I don't know why.. but every page refresh is requesting all data many times ..and all per groups... let me explain me with my log... 09/01/07 15:32:10 index [0] pageSize [10] 09/01/07 15:32:10 index [10] pageSize [10] 09/01/07 15:32:10 index [20] pageSize [10] 09/01/07 15:32:10 index [30] pageSize [10] 09/01/07 15:32:10 index [40] pageSize [10] 09/01/07 15:32:10 index [50] pageSize [10] 09/01/07 15:32:10 index [60] pageSize [10] 09/01/07 15:32:10 index [70] pageSize [10] 09/01/07 15:32:10 index [80] pageSize [10] 09/01/07 15:32:10 index [90] pageSize [10] 09/01/07 15:32:10 index [100] pageSize [10] 09/01/07 15:32:10 index [10] pageSize [10] 09/01/07 15:32:10 index [20] pageSize [10] 09/01/07 15:32:10 index [30] pageSize [10] 09/01/07 15:32:10 index [40] pageSize [10] 09/01/07 15:32:10 index [50] pageSize [10] 09/01/07 15:32:10 index [60] pageSize [10] 09/01/07 15:32:10 index [70] pageSize [10] 09/01/07 15:32:10 index [80] pageSize [10] 09/01/07 15:32:10 index [90] pageSize [10] 09/01/07 15:32:10 index [100] pageSize [10] 09/01/07 15:32:10 index [10] pageSize [10] 09/01/07 15:32:10 index [20] pageSize [10] 09/01/07 15:32:10 index [30] pageSize [10] 09/01/07 15:32:10 index [40] pageSize [10] 09/01/07 15:32:10 index [50] pageSize [10] 09/01/07 15:32:10 index [60] pageSize [10] 09/01/07 15:32:10 index [70] pageSize [10] 09/01/07 15:32:10 index [80] pageSize [10] 09/01/07 15:32:10 index [90] pageSize [10] 09/01/07 15:32:10 index [100] pageSize [10] 09/01/07 15:32:10 index [10] pageSize [10] 09/01/07 15:32:10 index [20] pageSize [10] 09/01/07 15:32:10 index [30] pageSize [10] 09/01/07 15:32:10 index [40] pageSize [10] 09/01/07 15:32:10 index [50] pageSize [10] 09/01/07 15:32:10 index [60] pageSize [10] 09/01/07 15:32:10 index [70] pageSize [10] 09/01/07 15:32:10 index [80] pageSize [10] 09/01/07 15:32:10 index [90] pageSize [10] 09/01/07 15:32:10 index [100] pageSize [10] I have just 107 records and 10 records per page, I'm not using rowselector.. is just a plane table with pagination... please help :'(

Lazy loading / Data on demand / Paging big tables solution

meeanaalenoor — GMT

How the LazyLoadableList Managed bean is processed in the .jspx page ?? Can you give me the code.

Lazy loading / Data on demand / Paging big tables solution

anil7734 — GMT

Can you provide the complete example for lazy loading, I am very new to IceFaces. Thanks

Re:Lazy loading / Data on demand / Paging big tables solution

jsfgeeks — GMT

hi, can anyone tell me where can i find this RowSelector.java in my project & which type dataModel is best to use i am currently using CachedRowSetSortableDataModel for database to bean binding. any help?

Re:Lazy loading / Data on demand / Paging big tables solution

Overwhelmer — GMT

Hi All, I have a similar problem of ybendek. It seems that the datatable is colling the get of all the elements. And this happens also when a change the page of the paginator, it reloads all the list. I don't use any rowselector. Anyone can help me?

Lazy loading / Data on demand / Paging big tables solution

flying_3x — GMT

Hi Guys, The lazy loading is really nice and work in my project. Thanks for your work. I have a small problem, as Lazy Loading only load the data in current page, is it possible to do column sort? I want to click the header of column and sort the data. Geoffrey

Re:Lazy loading / Data on demand / Paging big tables solution

priyasubu — GMT

Hi I hve a problem with lazy loading big tables we have successfully implemented loading large datasets and then paging them by chunking the datasets now the new problem I encounter is that in my datatable we have a popup component and when the popup component is triggered the whole datatable is refreshed and all the rows from the database are retrieved though I would like it only to refresh that page I am not sure where the problem coud be any help would be appreciated

Re:Lazy loading / Data on demand / Paging big tables solution

Tostis — GMT

Downloadable source code is different than code from page.