Does it make sense to swap between two QGraphicsView's if updating its content takes too long?

Does it make sense to swap between two QGraphicsView's if updating its
content takes too long?

This is a conceptual question. I want to avoid spending hours trying to do
something that will not work at all.
My setup:
I have 3 QGraphicViews (I would like to have more) with 1000s of elements.
Each element is a simple rectangle. However I need to dynamically add and
remove about a 1000 of those rectangles quite often. The elements that I
add and remove are never visible for the user (i.e. they are outside of
the sceneRect).
The problem:
This takes 70-100ms for each of the QGraphicsView. And as far as I
understand so far, I need to update them in the mainloop. This is painful,
as I run this update about every few seconds and the user experiences a
massive lag.
The Question:
Is it possible and will it really speed up the application, to operate on
two QGraphicViews each instead and doublebuffer them?
My idea:
My idea is to create two QGraphicViews for each one that I have now. Then
populate the first one normally while keeping the second one invisible.
Whenever I want to update the content of the QGraphicView I pass the
pointer of the second QGraphicView to a thread that does the update and
then just swap the visibility.
Now since both QGraphicView belong to the same QWindow, how will Qt handle
that? What do I need to take care of? And will it really be faster of will
it take a similar amount to swap between the QGraphicViews?
I am using PyQt4 if that makes any difference. (I do threading using
QThread sub-classes)
Thanks