HK1192343B - Drag and drop of objects between applications - Google Patents

Abstract

Methods, apparatuses and systems directed to capturing an object rendered on the first window of a display by a first program, extracting the object, permitting a user to drag the object across the display into a second window of the display containing a second program, and importing the object into the second program in substantially real-time. The drag and drop process occurs seamlessly to the user and permits a user to select one or more of a plurality of objects in one application, drag the object into a second application for modification, and drag the modified object back into the first application for real-time preview.

Description

Dragging and dropping of objects between applications

RELATED APPLICATIONSCross reference to

This application is U.S. provisional application No. 61/523,142, entitled "drag and drop of objects between applications," filed on 12/8/2011 and U.S. non-provisional application No. 13/571,182, filed on 9/8/2012, entitled "drag and drop of objects between applications," the entire contents of which are hereby incorporated by reference.

The present invention relates generally to importing or exporting three-dimensional graphical objects from a first application to a second application in real-time, and more particularly to importing/exporting graphical representations on a user's display between windows rendering the first and second applications.

Background

Typically, graphics programs render 2D or 3D objects by converting them into drawing instructions, which are then fed back to a graphics API (such as OpenGL or Direct 3D). Within the API rendering channel, the drawing instructions undergo various processing such as hidden surface removal, Z-caching, rasterization, clipping, and so forth, before being output as a 2D image on the application user display. Since the program data is to be disassembled to retrieve the OBJ file or other readable 3D format, it is often a difficult process to derive specific 3D objects from the graphics program, if possible. Similarly, importing a file into a 3D graphics program requires the assembly of 3D objects into the format required by the graphics program, and often requires repackaging the entire 3D object library for successful object import.

Disclosure of Invention

The present invention generally relates to exporting objects from a first 3D program for real-time rendering in a second 3D program. In one embodiment, a computer system hosts a plurality of application instances, each application instance corresponding to a local client application. The computer system simultaneously renders graphical output of application instances corresponding to at least two native client applications in separate windows on a computer system display using resources of a graphics processing unit of the computer system. A user seeks to export a 3D object from the first application, selects an object from the first window and drags the object to the second window. When the user drags the object, the object is rendered on the computer display according to the dragging instruction of the user. The user then places the object rendered in the second window into the second application, and the object is imported into the second application in real-time.

In one embodiment, the first computer system hosts a first application locally and the second application is hosted on an application server. The computer system renders the local application and the remote application in two separate windows on the computer system display through its local hardware and rendering APIs. The user seeks to export a 3D object from the first application, selects an object from the first window and drags the object to the second window. When the user drags the object, the object is rendered on the computer display according to the dragging instruction of the user. The user then places the object rendered in the second window in the second application and the object is imported into the second application in real-time.

In one embodiment, a first computer system hosts a first application locally and a second application is hosted on an application server with server-side rendering. The computer system renders a local application with its own graphics processor and graphics API, and the remote application is rendered by a server-side graphics API. A user seeks to export a 3D object from the first application, selects an object from the first window and drags the object to the second window. When the user drags the object, the object is rendered on the computer display according to the dragging instruction of the user. The user then places the objects rendered in the second window into the second application, and the objects are imported into the second application in real-time.

In one embodiment, a method for importing an object into a second application is disclosed. The method comprises the following steps: a first user input is received by a processor and an object rendered in a first window of a display by a first application and a rendering API is selected by the processor in response to the first user input. The method also includes extracting, via an engine, the object from the first application and receiving, by the processor, a second user input. In response to the second user input, the method comprises: dragging, by the processor, an object on the display from the first window to a second application rendered in a second window, and displaying, by the processor, the object in an intermediate space between the first window and the second window during the dragging. In response to the pair crossing the focus boundary of the second window, the method includes importing the object into the second application.

In an embodiment, selecting an object according to the method comprises: bypassing, by the processor, the first user input to the engine, intercepting, by the processor, a drawing command from the first application to the rendering API, and determining, by the processor, the object of the drawing command from the first application. The method further comprises the following steps: selecting, by the processor, the object according to a selection algorithm. In an embodiment, the selection algorithm is configured to select all objects related to the first object that is hit by the ray. In an embodiment, the selection algorithm is configured to select all objects having the same object identifier as the first object that is hit by the ray. In an embodiment, the selection algorithm is configured to select all objects having the same motion vector as the first object hit by the ray. In an embodiment, the selection algorithm is configured to select all objects having the same texture as the first object hit by the ray.

In an embodiment, the first user input selection object is a cursor selection area from a pointing device. In an embodiment, the first user input selecting an object comprises a user tracing a border around the object. In an embodiment, the first user input selection object comprises a selection tool that selects all adjacent pixels of the predetermined feature set. In an embodiment, the first user input selection object is a tap on a touch interface. In an embodiment, the first user input selection object refers to a gesture on a touch interface.

In an embodiment, the method of determining, by the processor, an object from the drawing command further comprises: assigning, by the processor, a camera on a near plane of a scene of the first user-input coordinates, and projecting, by the processor, a ray from the camera to a far plane and selecting a first object on which the ray hits. The method further comprises the following steps: the processor also receives user input to expand or filter a selection region, wherein expanding or filtering the selection region includes selecting or deselecting, by the processor, other objects in the scene that are related to the selected object.

In an embodiment, expanding or filtering the selection comprises: selecting or deselecting, by the processor, other objects in the scene having the same object identifier as the selected object. In an embodiment, expanding or filtering the selection comprises: selecting or deselecting, by the processor, other objects in the scene having the same motion vector as the selected object. In an embodiment, expanding or filtering the selection comprises: selecting or deselecting, by the processor, other objects in the scene having the same texture as the selected object. In an embodiment, expanding or filtering the selection comprises: selecting or deselecting, by the processor, other objects in the scene specified by further user input. In an embodiment, the specifying comprises receiving, by the processor, a user input, assigning, by the processor, a camera on a near plane of a scene of coordinates of the user input, and projecting, by the processor, a ray from the camera to a far plane and specifying a first object on which the ray hits.

In an embodiment, the processor dragging the object on the display comprises: the processor renders a boundless window and a selection in the boundless window, wherein the selection includes an object selected by the user. In an embodiment, in response to receiving user input to drag the borderless window from the first window to the second window, the method includes moving, by the processor, the borderless window across the display in accordance with the user input.

In an embodiment, a method for rendering, by the processor, the selection in the borderless window comprises: copying, by the processor, drawing commands related to the selection from the first application, the processor inserting drawing commands from the first application into a rendering API channel, and rendering, by the processor, the drawing commands via the rendering API.

In an embodiment, the method of importing the selection into the second application includes: converting, by the processor, a selection for implementation into the second application, and the processor rendering the selection in the second window via the engine during the converting. In an embodiment, converting the selection comprises: modifying, by the processor, the drawing command into a file format used by the second application. In an embodiment, the file format is an OBJ file.

Upon completion of the conversion, the method includes: importing, by the processor, the selection into the second application. Based on importing the object into the second application, the method further comprises: pausing, by the processor, the engine rendering process, and rendering, by the processor, the object from within the second application.

In an embodiment, the method of rendering the selection via the engine comprises: inserting, by the processor, a drawing command into a rendering API channel operable to instruct the rendering API to render the selection into the second window. In an embodiment, the second application has its own rendering API, and rendering the selection from within the second application comprises: the processor renders the selection in the second window using a rendering API of the second application.

In an embodiment, a method of rendering a selection in the borderless window comprises: obtaining, by the processor, a first condition and a second condition, the first condition including light and an environmental effect from the first application, the second condition including light and an environmental effect from the second application. The method further includes the processor gradually applying the first and second conditions according to the distance of the borderless window to the first and second windows.

In an embodiment, a system for exporting and importing an object from a first application to a second application is disclosed. In an embodiment, the object is a three-dimensional object. The system includes a graphics processing unit, a processor, and a storage medium having tangibly stored thereon program logic for execution by the processor. In an embodiment, the storage medium may further include one or more of the first and second applications. The program logic in the storage medium includes first user input receiving logic executed by a processor for receiving first user input. Selection logic included in the storage medium and executed by the processor to select an object rendered by a first application and a rendering API in a first window of a display in response to receiving the first user input. Extracting, by extraction logic embodied on the storage medium, the object from the first application. Further, the processor executes second user input receiving logic to receive a second user input, in response to receiving the second user input, drag logic to drag the object on the display from the first window to a second application rendered in the second window, and in response to the object crossing a focus boundary of the second window, the processor executes import logic contained in the storage medium for importing the object into the second application.

In an embodiment, the selection logic executed by the processor to select an object further includes bypass logic also executed by the processor to bypass the first user input from the first application. The selection logic further includes interception logic executed by the processor to intercept drawing commands applied to the rendering API from the first application, determination logic executed by the processor to determine the object from the drawing commands associated with the first user input, and selection logic executed by the processor to select the three-dimensional object and other objects according to a selection algorithm.

In an embodiment, the determination logic further comprises assignment logic executed by the processor to assign a camera in a near plane of the scene of coordinates input by the first user. The determining logic executed by the processor further includes ray casting logic to cast rays from the camera to the far plane and select a first object on which the rays hit.

In an embodiment, the drag logic executed by the processor includes window rendering logic to render a boundless window, selection rendering logic to render a selection in the boundless window that includes a user selected object, and movement logic to move the boundless window across the display in accordance with user input in response to receiving user input dragging the boundless window from the first window to the second window.

In an embodiment, the region rendering logic executed by the processor further includes copy logic to copy the drawing commands associated with the region, insert logic to insert the drawing commands into the rendering API channel, and drawing command rendering logic to render the drawing commands through the rendering API. In an embodiment, the selection rendering logic further comprises first condition obtaining logic executed by the processor to obtain a first condition comprising the illumination and environmental effect from the first application and second condition obtaining logic executed by the processor to obtain a second condition comprising the illumination and environmental effect from the second application. Further, the selection rendering logic executed by the processor may further comprise condition applying logic to apply the first and second conditions step by step based on distances of the windowless boundary from the first and second windows.

In an embodiment, the import logic executed by the processor further comprises: conversion logic to convert the selection for implementation to the second application such that the selection is imported to the second application based on completion of the conversion process; rendering logic for rendering the selection in the second window during the conversion process; and pause logic to pause the engine rendering process and render objects from within the second application based on importing the objects into the second application. In an embodiment, the conversion logic executed by the processor in the conversion process further comprises modification logic to modify the drawing commands into a file format used by the second application. In an embodiment, the file format is an OBJ file. In an embodiment, the rendering logic further comprises insertion logic, executed by the processor, to insert drawing commands into a rendering API channel operable to instruct the rendering API to render the selection to the second window. In an embodiment, a second application rendering API renders the selection based on importing the object into the second application in the second window.

A computer-readable storage medium having instructions stored thereon, which when executed by a processor, cause the processor to receive a first user input and render an object, rendered by a first application and a rendering API, in a first window of a display in response to the first user input. The instructions also cause the processor to extract, by an engine, the object from the first application. Further, the storage medium includes instructions to receive a second user input and drag the object on the display from the first window to a second application rendered in a second window in response to the second user input. The storage medium further includes instructions to import an object into the second application in response to the object crossing a focus boundary of the second window to import the object into the second application.

These and other embodiments in which features may be combined will be apparent to one of ordinary skill in the art by reference to the following detailed description and accompanying drawings.

Drawings

In the drawings, which are not to scale, like reference numerals designate like parts throughout the several views:

FIG. 1 illustrates an example of a computer system hosting two native applications and exporting a 3D object from a first application for importing a second application;

FIG. 2 illustrates an overall flow of the export and import process, including grabbing an object from a first application, dragging the object from the first application to a second application, and dropping the object in the second application for rendering;

FIG. 3 illustrates the flow of the grab process;

FIG. 4 illustrates a flow of a drag process;

FIG. 5 illustrates the flow of the drop process;

FIG. 6 illustrates a process flow for a reentry process;

FIG. 7 illustrates a representation of a computer system display performing the rendering process in conjunction with an environmental effect;

FIG. 8 illustrates an example of a computer system hosting a local application and an application server hosting a remote second application;

FIG. 9 illustrates an example of a computer system hosting a local application and an application server with server-side rendering hosting a remote second application;

FIG. 10 illustrates an exemplary computer system 1000 suitable for implementing one or more portions of a particular embodiment.

Detailed Description

The inventive subject matter is described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments. The inventive subject matter, however, may be embodied in many different forms and, thus, covered or claimed subject matter is not intended to be limited to any specific embodiment set forth herein; the particular embodiments provided are illustrative only. Likewise, a reasonably broad range of claimed or covered subject matter is contemplated. In addition, for example, the subject matter may be embodied as a method, apparatus, component, or system. Accordingly, embodiments may take forms such as hardware, software, firmware, or any combination thereof (except software itself). The following detailed description is, therefore, not to be taken in a limiting sense.

In the drawings, some features are exaggerated to show details of particular components (any dimensions, materials, and similar details shown in the figures are intended to be illustrative and not limiting). Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the disclosed embodiments.

The present invention is described below with reference to block diagrams and operational illustrations of methods and apparatus for selecting and presenting media related to a particular topic. It will be understood that each block of the block diagrams or operational illustrations, and combinations of blocks in the block diagrams or operational illustrations, can be implemented by analog or digital hardware and computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implement the functions/acts specified in the block diagrams or operational block or blocks. Various aspects or features will be presented in terms of systems that may include a number of devices, components, modules, and the like. It is to be understood and appreciated that the various systems may include additional devices, components, modules, etc. and/or may not include all of the devices, components, modules etc. set forth in the associated figures. Combinations of these approaches may also be used.

In some alternative implementations, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.

For the purposes of the present invention, the term "server" should be understood to refer to a service point that provides processes, databases, communications facilities. By way of example, and not limitation, the term "server" may refer to a single physical processor with associated communications and data storage and database facilities, or it may refer to an aggregation of networked or clustered processors, associated networks and storage devices, and operating on software and one or more database systems and application software supporting the services provided by the server.

The computer device may transmit or receive signals over a wired or wireless network, for example, or may process or store signals in a physical storage state, such as a memory, and may thus operate as a server. Thus, a device capable of operating as a server may include, for example, a dedicated rack-mounted server, a desktop computer, a laptop computer, a set-top box, an integrated device that combines various features, such as two or more of the features of the device described above, and so on. Servers may vary widely in configuration or performance, but typically may include one or more central processing units and memory. The Server also includes one or more mass storage devices, one or more power supplies, one or more wired or wireless network interfaces, one or more input/output interfaces, or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and so forth.

Throughout the specification and claims, terms may have meanings implicit or implicit in context beyond the meanings explicitly described. Likewise, the phrase "in one embodiment" as used herein does not necessarily refer to the same embodiment, and the phrase "in another embodiment" as used herein does not necessarily refer to a different embodiment. It is intended that, for example, the claimed subject matter includes combinations of exemplary embodiments in whole or in part. In general, terms may be understood, at least in part, from the context of their use. For example, terms such as "and," "or," or "and/or" as used herein may include a variety of meanings that depend, at least in part, on the context in which such terms are used. Typically, if "or" is used to associate a list such as A, B or C, it is intended herein to mean A, B and C when used in an inclusive sense and to mean A, B or C when used in an exclusive sense. Furthermore, the term "one or more" as used herein may be used in the singular to describe any capability, structure, or characteristic, or may be used in the plural to describe a combination of capabilities, structures, or characteristics, depending at least in part on the context. Also, terms such as indefinite article (a/an) or definite article (the) may be understood to imply the singular or the plural depending on the context. Moreover, the term "based on" may be understood as not necessarily intended to convey a specific set of factors, but rather, to allow for the presence of additional factors not necessarily explicitly described, depending, at least in part, on the context.

The present invention relates generally to exporting objects from a first 3D program for real-time rendering in a second 3D program. In one embodiment, a computer system hosts a plurality of application instances, each application instance corresponding to a local client application. The computer system simultaneously renders graphical output corresponding to application instances of at least two local client applications in separate windows on a computer system display using resources of a graphics processing unit of the computer system. A user seeking to export a 3D object from a first application selects an object from a first window and drags the object to a second window. When the user drags the object, the object is rendered on the computer display according to the dragging instruction of the user. The user then places the object in a second application that is rendered in a second window, and the object is imported into the second application in real-time.

Rendering can be considered the process of generating an image from a model, usually by means of a computer program. A model is typically a description of a three-dimensional (3D) object and may be represented in a well-defined language or data structure. The model may contain geometry, viewpoint, texture, lighting, shading, motion, and other suitable types of information. The image into which the model is rendered may be a digital image formed by a collection of many pixels or a raster graphics image. The present invention extends the rendering concept to generating images that represent arbitrary output for any application. Rendering may be performed based on arbitrary data, including two-dimensional (2D) data and 3D data. In addition to generating images from 3D models, particular embodiments may render images that represent application output, such as, by way of example and not limitation, web browsing applications, word processing applications, spreadsheet applications, multimedia applications, scientific and medical applications, and gaming applications.

Modifying objects from 3D programs is typically a difficult task, if possible. If the user does not have the original OBJ or other format file for modification in a 3D graphics program (such as 3D Studio Max or Maya), the user must decompile the 3D graphics file used by the first 3D program. The graphics file may be stored in a given directory under the program installation path or compiled into the actual program code itself. In any case, the user must run several steps to obtain the object file in a format that can be read by the 3D graphics program. Also, after modifying the object file, in order to view the appearance of the 3D object from inside the first program, the user must recompile or import the object into the code of the first program. This process is time consuming and more severe when using remote applications.

Since the rendering process is often resource demanding, it can be very computationally intensive, especially when the rendered image is of high resolution and high quality, rendering can be a type of task suitable for execution by a server. In the past, older computer systems required hours or days to render a three-dimensional model into a single 2D image. With the development and advancement of computer hardware, and particularly computer hardware specifically designed for computer graphics applications (such as gaming, multimedia, entertainment, or graphics), current computer systems can render each image in seconds or milliseconds. In fact, rendering a model as a single image typically does not occupy all available resources of the server. Thus, remote applications that utilize server-side rendering are becoming more popular.

To make it easier for 3D objects derived from a first 3D program to be imported into a second 3D program, the software engine may respond to user commands to select specific objects by intercepting drawing commands from a first application to a 3D graphics rendering channel and inserting them into the drawing commands for a given scene of the second application. In particular embodiments, the second application may be a remote application hosted on a separate server. In other embodiments, the second application may be a remote application with server-side rendering.

Fig. 1 illustrates an exemplary computer system 101 running a local first application 105 and a local second application 106. In normal operation, the user activates the system 101, for example by controlling the user hardware 108, and the I/O interface 107 converts the signal from the user/hardware 108 into instructions to the first application 105 or the second application 106. Applications 105 and 106 each output drawing commands to rendering API104 for rendering a 2D or 3D scene. Rendering API104 passes the drawing commands through a rendering channel (not shown) to convert the drawing commands into instructions for execution by graphics hardware 103 to render a 2D or 3D scene on display 102. In one embodiment, the first application 105 is rendered in a first window on a portion of the display 102 and the second application 106 is rendered in a second window on a different portion of the display 102. In an embodiment, engine 109 is a software program running on a processor (not shown) contained within system 101 having both first application 105 and second application 106. Engine 109 constantly monitors I/O interface 107 for instructions to initiate the drag and drop process. When these instructions are detected, the instructions bypass to engine 109 via path 110. The user may initiate the drag and drop process in a variety of ways including, but not limited to: a particular keystroke, a predetermined key or other pointer device input held with a mouse, a tap on a touch input device, or a particular gesture on a touch input device. Once the command has detoured to engine 109 via path 110, engine 109 allows the user to select a given object in any application window. The engine 109 also monitors the drawing commands from the first application 105 and the second application 106 to the rendering API104 and uses the circuitous user input 110 to determine which object or objects in the scene the user wishes to select. The engine 109 extracts drawing commands corresponding to the objects that the user wishes to select, and transmits them to the rendering API104 for rendering during the drag processing. During the drop process, the engine 109 continues to pass the drawing commands for the object to the rendering API104 for rendering in the window of the second application 106, but at the same time converts the drawing commands into a format for importing the second application 106. After the conversion and import process is completed, engine 109 stops sending drawing commands to rendering API104, and the selected object is exclusively rendered by second application 106. A more detailed explanation of the grab, drag and drop process will be given below. To simplify the discussion, only two applications are illustrated in FIG. 1. However, in practice it will be appreciated that computer system 101 may execute any number of applications in parallel to render various objects derived from one application to another in accordance with embodiments described herein.

FIG. 2 illustrates a high level flow of the drag and drop process. In step 201, a computer system begins running a plurality of applications. In step 202, the user initiates the grabbing process, which is described in detail in FIG. 4. In step 203, the user drags the desired object from the window displaying the first application to the window displaying the second application, as detailed in FIG. 5. Finally, in step 204, the user places the object in a window of a second application, also referred to as a reentry process, detailed in FIG. 6.

FIG. 3 illustrates the appearance of a user display during the drag and drop process. Initially, the display of the computer system contains two separate windows, where the first window 301 contains the rendered output of the first application and the second window 304 contains the rendered output of the second application. Objects 303a and 303b are rendered in the first window 301. In practice, the first window 301 and the second window 304 may contain any number of objects, and FIG. 3 is limited to two objects only for ease of discussion.

In fig. 3, a first window 301 is shown in an enlarged view as 303. The user selects object 303a in a variety of different ways, such as clicking on an input device or tapping the touchscreen at a single point 307 on object 303a, tracing a path 308 with the input device or on the touchscreen through object 303a, or drawing a marquee 306 around object 303a with the input device or on the touchscreen. Other input methods may be readily apparent to one of ordinary skill in the art including, but not limited to: gestures, a selection bar, and a polygon marquee.

Based on the selected object 303a, the user drags the object on the display along path 302 from the first window 301 to the second window 304. In some embodiments, the objects are copied and remain rendered in window 301 as copy 303c is rendered along path 302 in the intermediate space extending between first window 301 and second window 304. In other embodiments, real object 303a is moved from window 301 to window 304. Path 302 is determined by user input and can take any path from window 301 to window 304.

Based on the focused bezel across the second window 304, the engine initiates a resume or drop process. When the user has positioned object 303a in window 304 according to his or her requirements, the user initiates a command to drop object 303a into window 304. At that point, the drag and drop process is complete and the engine imports object 303a as object 303d into the second application for rendering in the second window 304.

Fig. 4 depicts a process flow of the grab process. In step 401, the engine receives a selection input selecting a desired object. As described above, the present invention provides a variety of select area input methods. Upon receiving the select input, the engine bypasses drawing commands from the first application to the rendering API to the engine itself. From these commands, the engine can recreate the scene rendered by the first application. In the context of a 3D scene, the engine has all 3D objects in a given scene at once, as well as the field of view and the camera points used by the first application to render the scene.

In one embodiment, the user input is a single point on the desired object on the first window. In step 402, the engine parses the input selection into a graphical object in the rendering display. To convert this two-dimensional input into a three-dimensional object, a conventional 3D object selection method is employed. One of these methods is to place a camera on the near plane of the 3D scene at the user-input location and cast light from the camera to the far plane, selecting the first object that the light hits. In another embodiment, the selection tool selects all objects that are in contact with the first object that the ray hits. In another embodiment, the selection tool selects all objects using the same object identifier (such as a tag or other metadata) as the first object that the ray hit. In another embodiment, the selection tool selects all objects that have the same texture as the first object that the ray hit. In yet another embodiment, the selection tool selects all objects that have the same motion vector as the first object hit by the ray.

In step 403, the engine filters or expands the user selection based on the user input. The user may choose to add the selected object in the same way as selecting the original object or some other input method, such as pressing a modifier to add a selection field and drawing a selection box around the other objects to be selected. Similarly, a pop-up window may be presented to the user to select other objects having the same motion vectors, textures, metadata, and so forth. Similarly, the user can filter out objects from the selection in a simulated manner. The user may have a button to subtract objects from the selection and click on individual objects or draw a marquee around the object to be removed from the selection. In addition, the user may be provided with a drop down menu to filter out objects with a given texture, motion vector, metadata tag, etc. The various methods of subtracting from or adding to the selection given by the present invention are well known to those of ordinary skill in the art.

After the user's requirements are met, the user enters a command to initiate a drag process in step 404. The commands may include, but are not limited to: pressing a button of the input device along with the mobile input device to drag a selected object, a particular gesture on the touch input device, pressing a key on a keyboard along with the mobile input device, and so on. The present invention may include a variety of methods of initiating the drag process 404.

Fig. 5 illustrates a processing flow of the drag processing. In step 501, the engine creates a window on a computer system display. In an embodiment, the window may have a visible border. In an embodiment, the window is borderless. The borderless window is simply a designated area on the second display that is completely transparent, and the only objects actually rendered on the display are the graphical objects contained within the borderless window. In step 502, the engine writes the object to the borderless window. The engine performs this step by bypassing the engine with drawing commands associated with the selection region from the first application. The engine then sends these drawing commands to the rendering API as objects to be rendered within the frameless window. The rendering API normally processes drawing commands through the rendering channel and renders the scene on the display within the borderless window. Because the borderless window is transparent, the object appears to move only as if it were from the first window to the second window. Thus, during the drag process, the rendering API always processes drawing commands from at least two applications, except the engine.

In accordance with a borderless window movement command from the user received via the input output interface 107, the engine transmits a drawing command associated with the selected object. As described, the present invention includes a variety of input methods for a user to adjust the position of a borderless window during the drag process.

At no time does the engine receive a user command to move the borderless window, the engine waits for a user input command to determine if a drop command has been issued in step 504. The drop command detection sends the process to the resume process in step 505. The drop command may be any command from the user input equipment to indicate that the user wishes to import the object into the second application rendered in the second window. The drop command may include, but is not limited to, releasing a button on a mouse or other input device, a gesture on a touch input device, or pressing another key on the input device. Other user input methods for drop commands will be apparent to those of ordinary skill in the art. In one embodiment, the reentry process begins when an object is dragged across the focused border of the second window.

Fig. 6 illustrates a process flow of the reentry process. In step 601, the resume process starts. The reentry process may be triggered by a user-defined drop command or the act of dragging the selection region across the second window focus bezel. When the reentry process begins, the engine begins to convert the objects from the drawing command into a format for implementation in the second application. In the 3D environment, the engine begins converting the drawing commands into a 3D object file for importing the second application. For example, a user may run a 3D gaming application in a first window and a 3D graphical editing program in a second window for editing a given model. Upon selecting the desired object, the user drags the object to a second window and the reentry process begins to convert drawing commands associated with the object into a 3D object file, such as an OBJ file.

In step 602, the engine continues to render the object by passing the drawing commands associated with the object to the rendering API. Because the conversion process is time consuming and processing intensive, the engine continues to render the objects as the conversion proceeds. The engine renders the object by inserting the drawing command into a stream of drawing commands from the second application to the rendering API. Thus, during the reentrant, the engine does not merely render the objects within the frameless window superimposed on top of the second window, but actually incorporates the objects into the second application as if the objects (including the environmental effects) were imported and rendered by the second application itself. Fig. 7 provides a detailed illustration of these features.

In step 603, the conversion is complete and the object file is imported to the second application. The import process is different for each application and each file format. In a 3D image editing program environment, files are imported into the program's workspace as if the user had opened the file directly from the 3D image editing program. In step 604, upon successfully importing the object into the second program, the engine stops its rendering of the object and the object is rendered by the second application. The entire reentry process occurs seamlessly without any indication to the user of multiple rendering processes or file transitions. The user is unaware of these background processes performed by the engine and the object is rendered as if it were simply dragged from the first window and dropped into the second window.

FIG. 7 depicts a representation of the drag and drop process in conjunction with environmental effects. In the first window 701, an object 702a (represented as a sphere for simplicity) is placed with a light source 703 from the top left of the window 701. In the second window 704, the environment of the 3D scene comprises a light source 705 from the upper right. During the crawling process, the engine takes the environmental effects and lighting of windows 701 and 704 and applies the environmental effects and lighting to the selected object 702a adaptively, depending on the distance of the object from each window. Thus, as object 702a is dragged towards the second window, the shading of object 702a varies according to the distance from light sources 703 and 705, as shown by representations 702b, 702c, 702d and 702 e. The engine renders the environmental effects by applying them to the drawing commands of the object before passing them to the rendering API. The environmental effect is not limited to only lighting, but it is also conceivable to apply fog, smoke, blur, particle effects, reflections and other well-known environmental effects to those skilled in the art.

FIG. 8 depicts an embodiment of a computer system 801 in which one of two applications is a remote application 810 running on an application server 809. In this case, the operation of the engine 805 is unchanged. The engine intercepts instructions from the I/O interface 807 and bypasses the engine along path 811 during the drag and drop process. Assuming that the user drags and drops from the native application 806 to the remote application 810, the drawing commands from the native application 806 to the rendering API804 are intercepted and used during the grabbing process to select the desired object for the user. During drag processing, engine 805 processes the rendering of objects in a borderless window by bypassing the drawing commands of the selection to rendering API 804. When the user places an object in the window of the remote application 810, the engine 805 begins the conversion process while continuing to render the selected object. Upon completion of the conversion, the converted object file is transmitted via network link 814 to application server 809 for import into remote application 810. After the import, the engine 805 stops transmitting drawing commands to the rendering API804, and the system operates normally. In another embodiment, the user drags the object from the remote application to the locally hosted application. The system operates in this configuration according to substantially the same mechanism.

FIG. 9 depicts an embodiment of a computer system 901 in which one application 908 is a remote application running on an application server 907, the application server 907 having server-side rendering through its own rendering API 909. For example, the user drags the application from a window of the local application 905 to a window of the remote rendering application 908. The operation of system 901 is substantially the same as that of fig. 8. Engine 910 intercepts I/O inputs from I/O interface 906 and bypasses them for use during the drag and drop process along path 914. During the grab process, the engine 910 bypasses drawing commands from the native application 905 to the native rendering API904 to the engine. After the user selects an object, the engine 910 detours the commands along path 915 to any of the local rendering APIs 904. During the drag process, the bypassed drawing commands for the selected object are rendered by the local rendering engine 904 to render the object in the borderless window. Upon initiation of the reentry process, the engine initiates a file conversion of the object to the object file for importing the remote application 908. When the file is converted, the file is imported to remote application 908 via path 916. The engine then stops rendering the objects through the local rendering API904, and the objects are rendered exclusively through the remote rendering API 909.

There are special cases in embodiments where the user wishes to select an object from a remote application with server-side rendering, such as application 908. In such an embodiment, the engine must access the output of the remote application 908 before it enters the remote rendering API 909. This must be a special implementation that requires software hosted on the remote application server 907 or, at a minimum, permission of the engine 910 by the server 907 to monitor the path between the remote application 908 and the rendering API 909. In this case, drawing commands from application 908 are bypassed to engine 910 over network connection 913. This special case only occurs when objects are grabbed from a remote application with server-side rendering.

The present invention encompasses a variety of settings, such as a change from dragging from one remote application to another, or copying/pasting objects from one application to multiple other applications. Such embodiments are readily contemplated by those of ordinary skill in the art. While the present invention describes a single instance of dragging and dropping from a first application to a second application, one skilled in the art can envision dragging from a first application to a second application, editing an object, dragging an edited object back to the first application to observe changes in real time.

Particular embodiments may be implemented by hardware, software, or a combination of hardware and software. By way of example, and not limitation, one or more computer systems may execute particular logic or software to perform one or more steps in one or more processes described and illustrated herein. One or more computer systems may be unitary or distributed, where appropriate across multiple computer systems or multiple data processing centers. The present invention applies to any suitable computer system. In particular embodiments, one or more steps of running one or more processes described and illustrated herein need not be limited to one or more particular geographic locations and need not have temporal limitations. By way of example, and not limitation, one or more computer systems may perform their functions in "real-time," "offline," or "batch mode," and may, where appropriate, be implemented in any suitable combination of the foregoing. One or more computer systems may perform one or more of their functions at different times, at different locations, using different processes, where appropriate. Reference herein to logic may encompass software and vice versa, where appropriate. Reference to software may encompass one or more computer programs, and vice versa, where appropriate. References to software may contain data, instructions, or both, and vice versa, where appropriate. Similarly, references to data may contain instructions, and vice versa, where appropriate.

One or more computer-readable storage media may store or otherwise contain software that implements particular embodiments. A computer-readable medium may be any medium that can carry, communicate, house, hold, retain, propagate, retain, store, transmit, transport, or otherwise contain software, where appropriate. A computer readable medium may be, for example, a biological, chemical, electronic, electromagnetic, infrared, magnetic, optical, quantum, or other suitable medium or a combination of two or more of the foregoing. A computer-readable medium may include one or more nanoscale elements or otherwise incorporate nanoscale design or fabrication. Exemplary computer readable storage media include, but are not limited to: compact Discs (CDs), Field Programmable Gate Arrays (FPGAs), floppy discs, hard discs, holographic storage devices, Integrated Circuits (ICs) such as Application Specific Integrated Circuits (ASICs), magnetic tapes, cache memories, Programmable Logic Devices (PLDs), Random Access Memory (RAM) devices, Read Only Memory (ROM) devices, semiconductor memory devices, and other suitable computer-readable storage media.

Software implementing particular embodiments may be written in any suitable programming language (which may be procedural or object oriented) or combination of programming languages, where appropriate. Any suitable type of computer system, such as a single-processor or multi-processor computer system, or multiple systems, may execute software implementing particular embodiments, where appropriate. Where appropriate, a general purpose computer system may execute software that implements a particular embodiment.

For example, FIG. 10 illustrates an exemplary computer system 1000 for implementing one or more portions of a particular embodiment. Although this disclosure describes and illustrates a particular computer system 1000 with particular elements in a particular configuration, the present disclosure applies to any suitable computer system with any suitable elements in any suitable configuration. Furthermore, computer system 1000 may have any suitable physical form, such as one or more Integrated Circuits (ICs), one or more Printed Circuit Boards (PCBs), one or more handheld or other devices (such as a mobile phone or PDA), one or more personal computers, or one or more supercomputers.

A system bus 1010 connects the various subsystems of the computer system 1000 to one another. Reference herein to a bus includes one or more digital signal lines that provide common functionality. The invention is applicable to any suitable system bus 1010, including any suitable bus structure having any suitable bus architecture, such as one or more memory buses, one or more peripheral buses, one or more local buses, or a combination of the above. Exemplary bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Enhanced ISA (EISA) bus, Micro Channel Architecture (MCA) bus, video electronics standards Association local (VLB) bus, Peripheral Component Interconnect (PCI) bus, PCI express bus (PCI-X), and Accelerated Graphics Port (AGP) bus.

The computer system 1000 includes one or more processors 1020 (or Central Processing Units (CPUs)). Processor 1020 may include cache 1022 for temporary local storage of instructions, data, or computer addresses. The processor 1020 is coupled to one or more memory devices, including a memory 1030. Memory 1030 may include Random Access Memory (RAM) 1032 and Read Only Memory (ROM) 1034. Data and instructions may be transferred bi-directionally between processor 1020 and RAM 1032. Data and instructions may be transferred uni-directionally from the ROM1034 to the processor 1020. RAM1032 and ROM1034 may include any suitable computer-readable storage media.

Computer system 1000 includes fixed storage 1040 bi-directionally coupled to processor 1020. Fixed storage 1040 may be connected to processor 1020 via storage control unit 1052. Fixed storage 1040 may provide additional data storage capacity and may include any suitable computer-readable storage media. Fixed storage 1040 may store an Operating System (OS) 1042, one or more executables 1044, one or more applications or programs 1046, data 1048, and the like. Fixed storage 1040 is typically a secondary storage medium (such as a hard disk) that is slower than primary storage. Where appropriate, information stored by fixed storage 1040 may be incorporated as virtual memory into memory 1030.

Processor 1020 may be connected to a variety of interfaces such as, for example, graphics control 1054, video interface 1058, input interface 1060, output interface 1062, and memory interface 1064, which in turn may be connected to appropriate devices, respectively. Exemplary input or output devices include, but are not limited to, a video display, a trackball, a mouse, a keyboard, a microphone, a touch-sensitive display, a card reader, a magnetic or paper tape reader, a tablet computer, a stylus, a voice or handwriting recognizer, a biometric reader, or a computer system. The network interface 1056 may connect the processor 1020 to another computer system or a network 1080. Through the network interface 1056, the processor 1020 may receive information from the network 1080 or send information to the network 1080 in the course of performing the steps of a particular embodiment. Particular embodiments may run solely on processor 1020. Particular embodiments may execute on processor 1020 and on one or more remote processors operating in conjunction.

In a network environment, where the computer system 1000 is connected to the network 1080, the computer system 1000 may communicate with other devices connected to the network 1080. The computer system 1000 may communicate with a network 1080 via a network interface 1056. For example, computer system 1000 may receive information from network 1080 in the form of one or more input packets (such as a request or response from another device) presented on network interface 1056, and memory 1030 may store the input packets for subsequent processing. Computer system 1000 can send information in the form of one or more outgoing packets (such as a request or response to another device) from network interface 1056 to network 1080, which can be stored by memory 1030 before being sent. The processor 1020 may access input or output packets in the memory 1030 to process them according to particular needs.

Computer system 1000 may have one or more input devices 1066 (which may include buttons, keyboard, mouse, stylus, etc.), one or more output devices 1068 (which may include one or more displays, one or more speakers, one or more printers, etc.), one or more storage devices 1070, and one or more storage media 1072. The input device 1066 may be external or internal to the computer system 1000. The output device 1068 may be external or internal to the computer system 1000. Storage 1070 may be external or internal to computer system 1000. The storage medium 1072 may be external or internal to the computer system 1000.

Particular embodiments include one or more computer storage products comprising one or more computer-readable storage media containing software for performing one or more steps in one or more processes described or illustrated herein. In particular embodiments, one or more portions of the media, software, or both may be specially designed and manufactured to perform one or more steps in one or more processes described or illustrated herein. In addition or as an alternative, in particular embodiments, one or more portions of the media, software, or both may be generic without specific design or manufacture of the processes described or illustrated herein. Exemplary computer-readable storage media include, but are not limited to, CDs (such as CD-ROMs), FPGAs, floppy disks, hard disks, holographic storage devices, ICs (such as ASICs), magnetic tape, caches, PLDs, RAM devices, ROM devices, semiconductor memory devices, and other suitable computer-readable storage media. In particular embodiments, the software may be machine code that a compiler may generate or one or more files containing high-level code that may be executed by a computer using a transcoder.

By way of example, and not limitation, memory 1030 may include one or more computer-readable storage media embodying software, and computer system 1000 may provide certain functions described or illustrated herein as a result of processor 1020 executing the software. Memory 1030 may store the software and processor 1020 may execute the software. The memory 1030 may read the software from a computer-readable storage medium of the mass storage device 1030 containing the software or from one or more other sources via the network interface 1056. When executing software, processor 1020 may perform one or more steps of one or more processes described or illustrated herein, which may include defining one or more data structures stored in memory 1030 and modifying the one or more data structures as directed by one or more portions of software, according to particular needs. In addition or as an alternative, the computer system 1000 may provide certain functions described or illustrated herein due to logic that is hard-wired or otherwise embodied in circuitry, which may operate in place of or in conjunction with software to perform one or more steps of one or more processes described or illustrated herein. The present invention may include any suitable combination of hardware and software, depending on particular needs.

In a particular embodiment, the computer system 1000 may include one or more Graphics Processing Units (GPUs) 1024. In a particular embodiment, GPU1024 may include one or more integrated circuits and/or processing cores directed to mathematical operations commonly used in graphics rendering. In some embodiments, the GPU1024 may use a specialized graphics unit instruction set, while in other embodiments, the GPU may use a CPU-like (e.g., modified x 86) instruction set. Graphics processing unit 1024 may perform a number of basic graphics operations, such as bitmap transfers, texture mapping, pixel shading, frame buffering, and so forth. In particular embodiments, GPU1024 may be a graphics accelerator, a General Purpose GPU (GPGPU), or any other suitable processing unit.

In a particular embodiment, GPU1024 may be included in a graphics or display card that is installed on the hardware system architecture via a card slot. In other embodiments, GPU1024 may be integrated onto a motherboard of a computer system architecture. Suitable graphics processing units may include AMD company (R) AMD R7XX based GPU devices (Radon (R) HD4 XXX), AMD R8XX based GPU devices (Radon (R) HD10 XXX), Intel company (R) Larabee based GPU devices (not published), nVidia company (R) 8 series GPUs, nVidia (R) 9 series GPUs, nVidia (R) 100 series GPUs, nVidia (R) 200 series GPUs, and any other DXll-capable GPUs.

Although the invention describes or illustrates particular operations occurring in a particular order, the invention is applicable to any suitable operations occurring in any suitable order. Further, the invention is applicable to any suitable operation being repeated one or more times in any suitable order. Although the invention describes or illustrates particular operations occurring in a particular order, the invention applies to any suitable operations occurring substantially simultaneously, where appropriate. Any suitable operation or sequence of operations described or illustrated herein may be interrupted, suspended, or controlled by another process, such as an operating system or kernel, where appropriate. The actions can be performed in an operating system environment or as a stand-alone program occupying all, or a substantial part, of the system processes.

The present invention encompasses all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments herein that a person having ordinary skill in the art would comprehend. Also, where appropriate, the appended claims encompass all changes, substitutions, variations, alterations, and modifications to the example embodiments herein that a person having ordinary skill in the art would comprehend.

To achieve the objects of the invention, a computer readable medium stores computer data in a machine readable form, where the data may include computer program code executable by a computer. By way of example, and not limitation, computer readable media may comprise computer readable storage media for tangible or fixed storage of data or communication media for temporal decoding of code-containing signals. Computer-readable storage media, as used herein, refers to physical or tangible memory (as opposed to signals) and includes, but is not limited to: volatile and nonvolatile, removable and non-removable media implemented in any method or technology for the tangible storage of information such as computer readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to: computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other physical or material medium which can be used to tangibly store the desired information or data or instructions and which can be accessed by a computer or processor.

Claims (16)

1. A method, comprising:

receiving, by a processor, a first user input;

selecting, by the processor, an object in response to the first user input, the object rendered in a first window of a display by a first application and a rendering API, wherein selecting the object comprises intercepting a drawing command from the first application to the rendering API;

extracting, by the processor, the object from the first application via an engine that monitors received user input;

receiving, by the processor, a second user input for dragging the object on the display from the first window tray to a second application rendered in a second window;

dragging the object from the first window to the second window in response to the second user input, including:

rendering, by the processor, a borderless window;

rendering, by the processor, a palette in the borderless window by bypassing the intercepted drawing commands to the engine, wherein the palette includes the object selected by the user;

moving, by the processor, the borderless window including the selection region across the display from the first window to the second window in accordance with the second user input;

responsive to the selection in the borderless window crossing a focused border of the second window, importing, by the processor, the selection in the borderless window to the second application, the importing including inserting the intercepted drawing commands into a rendering API channel operable to instruct the rendering API to render the selection to the second window; and

based on the importing, ceasing, by the processor, intercepting and bypassing the drawing commands from the first application to the rendering API.

2. The method of claim 1, wherein selecting an object further comprises:

bypassing, by the processor, the first user input to the engine;

determining, by the processor, the object from the drawing command; and

selecting, by the processor, the object and other objects according to a selection algorithm.

3. The method of claim 2, wherein determining, by the processor, the object comprises:

assigning, by the processor, a camera on a near plane of a scene of the first user-input coordinates;

projecting, by the processor, light from the camera to a far plane; and

selecting, by the processor, a first object on which the ray hits.

4. The method of claim 3, further comprising:

further user input is received by the processor to expand or filter the selection.

5. The method of claim 4, wherein expanding or filtering the selection comprises:

selecting or deselecting, by the processor, other objects in the scene that are related to the selected object.

6. The method of claim 4, wherein expanding or filtering the selection comprises:

selecting or deselecting, by the processor, other objects in the scene for which a process is specified by the further user input, wherein the specifying process comprises:

receiving, by the processor, another user input for one of object selection or deselection;

assigning, by the processor, another camera on a near plane of the scene at the other user-input coordinate; and

projecting, by the processor, a ray from the camera to the far plane and specifying the first object that the ray hits.

7. The method of claim 1, wherein rendering, by the processor, the selection in the borderless window comprises:

copying, by the processor, drawing commands associated with the selection from the first application;

inserting, by the processor, the drawing command from the first application into a channel of a rendering API;

rendering, by the processor, the drawing commands via the rendering API.

8. The method of claim 1, wherein rendering the selection in the borderless window comprises:

obtaining, by the processor, a first condition comprising lighting and environmental effects from the first application;

obtaining, by the processor, a second condition comprising lighting and environmental effects from the second application;

gradually applying, by the processor, the first condition and the second condition according to a distance from the first window and the second window to the borderless window.

9. The method of claim 1, wherein importing the selection into a second application comprises:

converting, by the processor, the selection for implementation into the second application;

rendering, by the processor, the selection via the engine in the second window during the transition;

importing, by the processor, the selection into the second application based on completion of the conversion; and

pausing, by the processor, the engine rendering process based on importing the object into the second application, and rendering, by the processor, the object from within the second application.

10. The method of claim 9, wherein converting the selection comprises:

modifying, by the processor, the drawing command to a file format employed by the second application.

11. The method of claim 10, wherein the second application has its own rendering API, and rendering the selection from within the second application comprises: rendering, by the processor, the selection in the second window using a rendering API of the second application.

12. A system, comprising:

a graphics processing unit;

a processor; and

a storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:

first user input receiving logic, executed by the processor, to receive a first user input;

selection logic, executed by the processor, to select a rendering object in a first window of a display through a first application and a rendering API in response to receiving the first user input;

extracting logic, executed by the processor, to extract the object from the first application via an engine that monitors the received user input, wherein selecting an object comprises intercepting a drawing command from the first application to the rendering API;

second user input receiving logic, executed by the processor, to receive a second user input;

drag logic executed by the processor for dragging the object on the display from the first window to a second application rendered in a second window in response to receiving the second user input, the drag logic further comprising:

window rendering logic executed by the processor to render a borderless window;

select rendering logic executed by the processor to render a select in the borderless window by bypassing the intercepted drawing commands to the engine, wherein the select includes the object selected by the user; and

movement logic executed by the processor to, in response to receiving the second user input dragging the borderless window from the first window to the second window, move the borderless window across the display from the first window to the second window in accordance with the second user input;

importing logic, executed by the processor, to import the selection in the boundless window into the second application in response to the selection in the boundless window crossing a focused bezel of the second window, the importing comprising inserting the intercepted drawing commands into a rendering API channel operable to instruct the rendering API to render the selection to the second window; and

stopping logic, executed by the processor, to stop intercepting and bypassing the drawing commands from the first application to the rendering API based on the importing.

13. The system of claim 12, wherein the selection logic executed by the processor to select an object further comprises:

detour logic, executed by the processor, to detour the first user input from the first application;

determining logic, executed by the processor, to determine the object from the drawing command associated with the first user input; and

selection logic, executed by the processor, to select the object and other objects according to a selection algorithm.

14. The system of claim 12, wherein determining, by the processor, the object comprises:

assigning logic, executed by the processor, to assign a camera on a near plane of a scene of the first user-input coordinates; and

ray casting logic, executed by the processor, to cast rays from the camera to a far plane and select a first object on which the rays hit.

15. The system of claim 12, wherein the import logic further comprises:

conversion logic, executed by the processor, to convert the selection for implementation into the second application, causing the selection to be imported into the second application based on completion of the conversion;

rendering logic, executed by the processor, to render the selection in the second window during the conversion process; and

pause logic, executed by the processor, to pause the engine rendering process and render the object from within the second application based on importing the object into the second application.

16. The system of claim 12, wherein the selection rendering logic further comprises:

first condition acquisition logic executed by the processor to acquire a first condition comprising illumination and environmental effects from the first application;

second condition acquisition logic executed by the processor to acquire a second condition comprising illumination and environmental effects from the second application; and

conditional applying logic, executed by the processor, to gradually apply the first condition and the second condition according to a distance from the first window and the second window to the unbounded window.